This paper reports on a novel verification and performance evaluation framework specifically designed and developed to facilitate a standardized comparative performance evaluation for commercial detection, tracking and identification (DTI) solutions to counter Unmanned Aerial System (UAS) threats. The test methodology is designed to compare commercial systems in a fair and reproducible manner based on end-user defined criteria.
DTI systems are increasingly relevant for e.g., perimeter protection of military facilities, critical infrastructures and public events and the expected end-users are law enforcement agencies, the military, civil defense agencies and private entities. However, such systems are commonly hard to benchmark in a fair and comparable manner and performance claims of these systems are currently not supported by evidence. In addition, no standardized test methodologies are currently available making it near impossible to compare competing DTI systems.
In Courageous we developed an objective driven test methodology for use by the civilian sector. Courageous leads to a comparative performance evaluation system for commercial DTI solutions for Counter-UAS systems (C-UASs) using operationally relevant end-user scenarios and a generic DTI system lay-out. The work takes into account contextual information as well as end-user input, albeit focusing primarily on civilian use cases so far. We outline the process taken as well as the resulting system and discuss how the systems should be evaluated and validated iteratively over time. We furthermore elicit end-user input from the defense domain and argue that the scope of Courageous should be broadened to include military challenges, aspects and concerns.
The work with regard to homeland security use-cases, presented here, has firstly been verified in a simulation environment where a number relevant scenarios were used and the output of the simulation injected into the testing system. Validation of the work in a relevant environment has been done in three operational trials.
The results from the operational trials held for homeland security scenarios show that the method allows for performance evaluation at component level (i.e., detection, tracking or identification component) and at system level (combinations of these components and integrated DTI system of system solutions).
We present a study of border surveillance systems for automatic threat estimation. The surveillance systems should allow border control operators to be triggered in time so that adequate responses are possible. Examples of threats are smuggling, possibly by using small vessels, cars or drones, and threats caused by unwanted persons (e.g. terrorists) crossing the border. These threats are revealed by indicators which are often not exact and evidence for these indicators incorporates significant amounts of uncertainty. This study is linked to the European Horizon 2020 project ALFA, which focuses on the detection and threat evaluation of low flying objects near the strait of Gibraltar. Several methods are discussed to fuse the indicators while taking the uncertainty into account, including Fuzzy Reasoning, Bayesian Reasoning, and Dempster-Shafer Theory. In particular the Dempster-Shafer Theory is elaborated since this approach incorporates evaluation of unknown information next to uncertainty. The method is based on belief functions representing the indicators. These functions show a gradual increase or decrease of the suspiciousness depending on input parameters such as object speed, size etc. The fusion methods give two output values for each track: a suspect probability and an uncertainty value. The complete dynamic risk assessment of detected flying objects is evaluated by the automatic system and targets with probabilities exceeding a certain threshold and appropriate uncertainty values are presented to the border control operators.
Access to the requested content is limited to institutions that have purchased or subscribe to SPIE eBooks.
You are receiving this notice because your organization may not have SPIE eBooks access.*
*Shibboleth/Open Athens users─please
sign in
to access your institution's subscriptions.
To obtain this item, you may purchase the complete book in print or electronic format on
SPIE.org.
INSTITUTIONAL Select your institution to access the SPIE Digital Library.
PERSONAL Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.