Rigorous evaluation of vector-associated model performance (REVAMP)

Sunshine Stacy; Patrick Fisher; Nicholas Del Rio

doi:10.1117/12.2586124

12 April 2021 Rigorous evaluation of vector-associated model performance (REVAMP)

Sunshine Stacy, Patrick Fisher, Nicholas Del Rio

Proceedings Volume 11746, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III; 117462W (2021) https://doi.org/10.1117/12.2586124
Event: SPIE Defense + Commercial Sensing, 2021, Online Only

Conference Poster

Abstract

DevOps engineers take many factors into account when assessing the suitability of AI/ML algorithms for operations. They rely on documentation about data requirements, parameter settings, theoretical limits, and operating characteristics to anticipate how well an algorithm will perform in the field prior to writing a single line of code. However, the USAF currently lacks code quality and documentation standards for AI/ML, which forces downstream consumers to make assumptions about the behavior of algorithms that often lead to unexpected results and wasted efforts. Therefore, we present a preliminary set of criteria by which to judge the maturity of word embedding algorithms in terms of reproducibility, testability, and documentation quality. We hope the criteria will grow into a set of quality requirements that govern how the DoD procures AI/ML capabilities, and eventually motivate the need for an ML development maturity model. Our nascent evaluation criteria surfaced during our own struggles with trying to replicate and compare the performance between a well-known word embedding algorithm (Word2Vec) and a custom graph embedding algorithm (IRI2Vec), procured specifically for the purpose of connecting missions across ATOs. We walk through two case studies in which we applied our evaluation criteria to assess the maturity of both Word2Vec and IRI2Vec

Conference Presentation

Citation Download Citation

Sunshine Stacy, Patrick Fisher, and Nicholas Del Rio "Rigorous evaluation of vector-associated model performance (REVAMP)", Proc. SPIE 11746, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications III, 117462W (12 April 2021); https://doi.org/10.1117/12.2586124

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available