Leveraging synthetic data for robust gesture recognition

Kapil Katyal; Rama Chellappa; Ketul Shah; Arun Reddy; Judy Hoffman; William Paul; Rohita Mocharla; David Handelman; Celso de Melo

doi:10.1117/12.2663827

13 June 2023 Leveraging synthetic data for robust gesture recognition

Kapil Katyal, Rama Chellappa, Ketul Shah, Arun Reddy, Judy Hoffman, William Paul, Rohita Mocharla, David Handelman, Celso de Melo

Author Affiliations +

Proceedings Volume 12529, Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications; 125290Y (2023) https://doi.org/10.1117/12.2663827
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Conference Poster

Abstract

Effective communication and control of a team of humans and robots is critical for a number DoD operations and scenarios. In an ideal case, humans would communicate with the robot teammates using nonverbal cues (i.e., gestures) that work reliably in a variety of austere environments and from different vantage points. A major challenge is that traditional gesture recognition algorithms using deep learning methods require large amounts of data to achieve robust performance across a variety of conditions. Our approach focuses on reducing the need for “hard-to-acquire” real data by using synthetically generated gestures in combination with synthetic-to-real domain adaptation techniques. We also apply the algorithms to improve the robustness and accuracy of gesture recognition from shifts in viewpoints (i.e., air to ground). Our approach leverages the soon-to-be released dataset called Robot Control Gestures (RoCoG-v2), consisting of corresponding real and synthetic videos from ground and aerial viewpoints. We first demonstrate real-time performance of the algorithm running on low-SWAP, edge hardware. Next, we demonstrate the ability to accurately classify gestures from different viewpoints with varying backgrounds representative of DoD environments. Finally, we show the ability to use the inferred gestures to control a team of Boston Dynamic Spot robots. This is accomplished using inferred gestures to control the formation of the robot team as well as to coordinate the robot’s behavior. Our expectation is that the domain adaptation techniques will significantly reduce the need for real-world data and improve gesture recognition robustness and accuracy using synthetic data.

Citation Download Citation

Kapil Katyal, Rama Chellappa, Ketul Shah, Arun Reddy, Judy Hoffman, William Paul, Rohita Mocharla, David Handelman, and Celso de Melo "Leveraging synthetic data for robust gesture recognition", Proc. SPIE 12529, Synthetic Data for Artificial Intelligence and Machine Learning: Tools, Techniques, and Applications, 125290Y (13 June 2023); https://doi.org/10.1117/12.2663827

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available