Implementation of a mixed-reality flight simulator: blending real and virtual with a video-see-through head-mounted display

Johannes Maria Ernst; Tim Laudien; Sven Schmerwitz

doi:10.1117/12.2664848

12 June 2023 Implementation of a mixed-reality flight simulator: blending real and virtual with a video-see-through head-mounted display

Johannes Maria Ernst, Tim Laudien, Sven Schmerwitz

Author Affiliations +

Proceedings Volume 12538, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V; 125380R (2023) https://doi.org/10.1117/12.2664848
Event: SPIE Defense + Commercial Sensing, 2023, Orlando, Florida, United States

Abstract

Conventional flight simulators usually include a complex and expensive outside vision projection system. Especially scenarios where the helicopter pilots look far to the side or through the windows near the pedals require a large projection dome to provide an image of the outside world. Additionally, simulators used for research need to be highly customizable: For rapid prototyping of new flight deck designs, the cockpit mockup must be adaptable enough to change the appearance and arrangement of its elements. The recent technological advancements of head-mounted displays (HMDs) offer many new ways to create a simulator that fulfills the stated requirements at moderate cost. A non-see-through HMD can immerse the pilots into a computer-generated cockpit with “unrestricted” virtual out-the-window view. The downwards view is even better than with dome projections. Such a fully virtual approach, however, requires complex finger-tracking and haptic feedback solutions to enable the user to interact with the cockpit. By contrast, a video-see-through HMD allows us to selectively combine a highly customizable virtual world with a video-stream of the real surroundings. One can, for instance, show the pilot’s hands and relevant parts of the physical flight deck mockup, enriched with virtual elements and virtual out-the-window view. In such a mixed setup, the pilots can naturally and directly interact with conventional input devices in an otherwise virtual environment. The paper presents our implementation of a mixed reality simulator with the Varjo XR-3 video-see-through HMD. We assess different variants, discuss implementation details like real-to-virtual-world-alignment, and explain the major challenges of such setups.

Conference Presentation

Citation Download Citation

Johannes Maria Ernst, Tim Laudien, and Sven Schmerwitz "Implementation of a mixed-reality flight simulator: blending real and virtual with a video-see-through head-mounted display", Proc. SPIE 12538, Artificial Intelligence and Machine Learning for Multi-Domain Operations Applications V, 125380R (12 June 2023); https://doi.org/10.1117/12.2664848

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available