research-article

3D Flight Planning Using Extended Reality

Authors:

Adrian Sarbach,

Martin RaubalAuthors Info & Claims

SUI '24: Proceedings of the 2024 ACM Symposium on Spatial User Interaction

Article No.: 5, Pages 1 - 10

https://doi.org/10.1145/3677386.3682088

Published: 07 October 2024 Publication History

Abstract

For all pilots, before getting on their aircraft, they plan their flight. Through this, they preview flight information and assess the flight situation. However, despite the three-dimensional nature of flying, the majority of planning processes still rely on 2D map-based software. Extended Reality (XR) technology has been widely used in aviation, in particular in pilot and crew member training. Yet, there have been very few studies or applications that incorporate 3D visualization and XR into flight planning. To address this gap, our research focuses on the implementation of a 3D flight planning application in XR. Through a user study (N=16), we demonstrate that flight planning in 3D XR outperforms traditional 2D environments in terms of user experience. The study also provides insights into the preferred method of route planning: creating flight routes through a series of waypoints has better usability compared to drawing flight routes freely in a 3D XR environment.

Supplemental Material

ZIP File

Supplementary material (vertical profiles and maps)

Download
7.16 MB

References

[1]

Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5643–5654. https://doi.org/10.1145/3025453.3025474

Digital Library

[2]

Doug A Bowman and Ryan P McMahan. 2007. Virtual Reality: How Much Immersion Is Enough?Computer 40 (July 2007), 36–43. Issue 7. https://doi.org/10.1109/MC.2007.257

Digital Library

[3]

John Brooke. 1996. SUS: A ’quick and dirty’ usability scale. In Usability Evaluation In Industry, Patrick W Jordan, Bruce Thomas, Ian Lyall McClelland, and Bernard Weerdmeester (Eds.). CRC Press, London, England, 207–212. https://doi.org/10.1201/9781498710411-35

[4]

Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc. T. Olano. 1992. 3DM: a three dimensional modeler using a head-mounted display. In Proceedings of the 1992 symposium on Interactive 3D graphics - SI3D ’92 (Cambridge, Massachusetts, USA) (SI3D ’92). ACM Press, New York, NY, USA, 135–138. https://doi.org/10.1145/147156.147182

Digital Library

[5]

Jamie Cross, Christine Boag-Hodgson, Tim Ryley, Timothy J. Mavin, and Leigh Ellen Potter. 2023. Using Extended Reality in Flight Simulators: A Literature Review., 3961-3975 pages. Issue 9. https://doi.org/10.1109/TVCG.2022.3173921

Digital Library

[6]

Michael F. Deering. 1995. HoloSketch: a virtual reality sketching/animation tool. ACM Transactions on Computer-Human Interaction 2, 3 (Sept. 1995), 220–238. https://doi.org/10.1145/210079.210087

Digital Library

[7]

Janez Demšar. 2006. Statistical Comparisons of Classifiers over Multiple Data Sets. J. Mach. Learn. Res. 7 (Dec. 2006), 1–30.

Digital Library

[8]

EASA. 2022. Annual Safety Review 2022. https://doi.org/10.2822/056444

[9]

Foreflight. 2024. ForeFlight. https://foreflight.com/products/foreflight-mobile/ Software.

[10]

Milton Friedman. 1937. The Use of Ranks to Avoid the Assumption of Normality Implicit in the Analysis of Variance. J. Amer. Statist. Assoc. 32, 200 (1937), 675–701. https://doi.org/10.1080/01621459.1937.10503522

[11]

Tanveer Gangabissoon, Girish Bekaroo, and Waseemah Moedeen. 2020. Application of augmented reality in aviation: improving engagement of cabin crew during emergency procedures training. In Proceedings of the 2nd International Conference on Intelligent and Innovative Computing Applications (Plaine Magnien, Mauritius). Association for Computing Machinery, New York, NY, USA, Article 32, 8 pages. https://doi.org/10.1145/3415088.3415120

Digital Library

[12]

Garmin Pilot. 2024. Garmin Pilot. Software, Version 24.5.0.

[13]

Águeda Gómez-Cambronero, Ignacio Miralles, Anna Tonda, and Inmaculada Remolar. 2023. Immersive Virtual-Reality System for Aircraft Maintenance Education: A Case Study. Applied Sciences 13 (May 2023), 5043. https://doi.org/10.3390/app13085043

[14]

Kyungjin Han, Syed Hammad Hussain Shah, and Jong Weon Lee. 2019. Holographic Mixed Reality System for Air Traffic Control and Management. Applied Sciences 9, 16 (Aug. 2019), 18 pages. https://doi.org/10.3390/app9163370

[15]

Takeo Igarashi and John F. Hughes. 2001. A suggestive interface for 3D drawing. In Proceedings of the 14th annual ACM symposium on User interface software and technology (Orlando, Florida) (UIST ’01). ACM, New York, NY, USA, 173–181. https://doi.org/10.1145/502348.502379

Digital Library

[16]

Christopher Katins, Sebastian S. Feger, and Thomas Kosch. 2023. Exploring Mixed Reality in General Aviation to Support Pilot Workload. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI EA ’23). Association for Computing Machinery, New York, NY, USA, Article 116, 7 pages. https://doi.org/10.1145/3544549.3585742

Digital Library

[17]

Christopher Katins, Sebastian Stefan Feger, and Thomas Kosch. 2023. Pilots’ Considerations Regarding Current Generation Mixed Reality Headset Use in General Aviation Cockpits. In Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia (Vienna, Austria) (MUM ’23). Association for Computing Machinery, New York, NY, USA, 159–165. https://doi.org/10.1145/3626705.3627785

Digital Library

[18]

Kevin Kim, Tiffany C. K. Kwok, Sailin Zhong, Peter Kiefer, and Martin Raubal. 2024. Can You Sketch in 3D? Exploring Perceived Feasibility and Use Cases of 3D Sketch Mapping. In 16th International Conference on Spatial Information Theory (COSIT 2024)(Leibniz International Proceedings in Informatics (LIPIcs)). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 18 pages. In Press.

[19]

Kevin Gonyop Kim, Jakub Krukar, Panagiotis Mavros, Jiayan Zhao, Peter Kiefer, Angela Schwering, Christoph Hölscher, and Martin Raubal. 2022. 3D Sketch Maps: Concept, Potential Benefits, and Challenges. In 15th International Conference on Spatial Information Theory (COSIT 2022)(Leibniz International Proceedings in Informatics (LIPIcs), Vol. 240). Schloss Dagstuhl – Leibniz-Zentrum für Informatik, Dagstuhl, Germany, 14:1–14:7. https://doi.org/10.4230/lipics.cosit.2022.14

[20]

Bettina Laugwitz, Theo Held, and Martin Schrepp. 2008. Construction and Evaluation of a User Experience Questionnaire. In HCI and Usability for Education and Work, Andreas Holzinger (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 63–76. https://doi.org/10.1007/978-3-540-89350-9_6

Digital Library

[21]

Nuno Moura Lopes, Fátima Trindade Neves, and Manuela Aparicio. 2024. Key Insights from Preflight Planning for Safety Improvement in General Aviation: A Systematic Literature Review. Applied Sciences 14, 9 (2024), 29 pages. https://doi.org/10.3390/app14093771

[22]

Lufthansa Aviation Training. 2021. VR hub. Online. https://www.lufthansa-aviation-training.com/virtual-reality-hub Last accessed 9 August 2024.

[23]

Lufthansa Aviation Training. 2024. Crew Training with Apple Vision Pro. Online. https://www.lufthansa-aviation-training.com/-/lat-und-3spintraining-develop-ai-supported-crew-training Last accessed 9 August 2024.

[24]

Mayra D. Barrera Machuca, Paul Asente, Wolfgang Stuerzlinger, Jingwan Lu, and Byungmoon Kim. 2018. Multiplanes: Assisted Freehand VR Sketching. In Proceedings of the Symposium on Spatial User Interaction (Berlin, Germany) (SUI ’18). ACM, New York, NY, USA, 36–47. https://doi.org/10.1145/3267782.3267786

Digital Library

[25]

Mayra D. Barrera Machuca, Wolfgang Stuerzlinger, and Paul Asente. 2019. Smart3DGuides: Making Unconstrained Immersive 3D Drawing More Accurate. In 25th ACM Symposium on Virtual Reality Software and Technology (Parramatta, NSW, Australia) (VRST ’19). ACM, New York, NY, USA, Article 37, 13 pages. https://doi.org/10.1145/3359996.3364254

Digital Library

[26]

Philippe Meister, Jack Miller, Kexin Wang, Michael C. Dorneich, Eliot Winer, Lori J. Brown, and Geoffrey Whitehurst. 2022. Designing Three-Dimensional Augmented Reality Weather Visualizations to Enhance General Aviation Weather Education. IEEE Transactions on Professional Communication 65, 2 (June 2022), 321–336. https://doi.org/10.1109/tpc.2022.3155920

[27]

Philippe Meister, Kexin Wang, Michael C. Dorneich, Eliot Winer, Lori Brown, and Geoff Whitehurst. 2022. Evaluation of the Effectiveness of Augmented Reality Enhanced Aviation Weather Training. In AIAA AVIATION 2022 Forum. American Institute of Aeronautics and Astronautics, Chicago, IL, USA, 1–13. https://doi.org/10.2514/6.2022-3780

[28]

National Centers for Environmental Information (NCEI). 2024. Global Forecast System (GFS). Online. https://www.ncei.noaa.gov/products/weather-climate-models/global-forecast Last accessed 9 August 2024.

[29]

Peter Björn Nemenyi. 1963. Distribution-free Multiple Comparisons. Ph. D. Dissertation. Princeton University.

[30]

NTSB. 2023. US Civil Aviation Accident Dashboard: 2008-2022. Online. https://www.ntsb.gov/safety/StatisticalReviews/Pages/CivilAviationDashboard.aspx Last accessed 9 August 2024.

[31]

Matthias Oberhauser and Daniel Dreyer. 2017. A virtual reality flight simulator for human factors engineering. Cognition, Technology & Work 19 (2017), 263–277. Issue 2. https://doi.org/10.1007/s10111-017-0421-7

Digital Library

[32]

Chao-Chung Peng, Ai-Chi Chang, and Yu-Lun Chu. 2022. Application of Augmented Reality for Aviation Equipment Inspection and Maintenance Training. In 2022 8th International Conference on Applied System Innovation (ICASI) (Nantou, Taiwan). IEEE, New York, NY, USA, 58–63. https://doi.org/10.1109/ICASI55125.2022.9774490

[33]

Denise Rey and Markus Neuhäuser. 2011. Wilcoxon-Signed-Rank Test. In International Encyclopedia of Statistical Science, Miodrag Lovric (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg, 1658–1659. https://doi.org/10.1007/978-3-642-04898-2_616

[34]

Jeff Sauro and James R. Lewis. 2012. Standardized Usability Questionnaires. Morgan Kaufmann, Boston, MA, USA, Chapter 8, 185–240. https://doi.org/10.1016/B978-0-12-384968-7.00008-4

[35]

Martin Schrepp, Jörg Thomaschewski, and Andreas Hinderks. 2017. Design and Evaluation of a Short Version of the User Experience Questionnaire (UEQ-S). International Journal of Interactive Multimedia and Artificial Intelligence 4 (Dec. 2017), 103–108. Issue 6. https://doi.org/10.9781/ijimai.2017.09.001

[36]

SkyDemon. 2024. SkyDemon. Software, Version 3.16.15.

[37]

Swisstopo. 2024. Digital Aeronautical Chart ICAO. Online. https://www.swisstopo.admin.ch/en/digital-aeronautical-chart-icao Last accessed 9 August 2024.

[38]

Swisstopo. 2024. Height Model DHM25 200m. Online. https://www.swisstopo.admin.ch/en/height-model-dhm25-200m Last accessed 9 August 2024.

[39]

Swisstopo. 2024. SWISSIMAGE 10 cm. Online. https://www.swisstopo.admin.ch/en/orthoimage-swissimage-10 Last accessed 9 August 2024.

[40]

Philipp Wacker, Adrian Wagner, Simon Voelker, and Jan Borchers. 2018. Physical Guides: An Analysis of 3D Sketching Performance on Physical Objects in Augmented Reality. In Proceedings of the Symposium on Spatial User Interaction (Berlin, Germany) (SUI ’18). ACM, New York, NY, USA, 25–35. https://doi.org/10.1145/3267782.3267788

Digital Library

[41]

Wen-Chung Wu and Van-Hoan Vu. 2022. Application of Virtual Reality Method in Aircraft Maintenance Service—Taking Dornier 228 as an Example. Applied Sciences 12 (2022), 7283. Issue 14. https://doi.org/10.3390/app12147283

[42]

Tianyi Xiao, Kevin Gonyop Kim, Jakub Krukar, Rajasirpi Subramaniyan, Peter Kiefer, Angela Schwering, and Martin Raubal. 2024. VResin: Externalizing Spatial Memory into 3D Sketch Maps. International Journal of Human-Computer Studies 190 (Oct. 2024), 103322. https://doi.org/10.1016/j.ijhcs.2024.103322

Digital Library

[43]

Min Xin, Ehud Sharlin, and Mario Costa Sousa. 2008. Napkin sketch: handheld mixed reality 3D sketching. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology (Bordeaux, France) (VRST ’08). ACM, New York, NY, USA, 223–226. https://doi.org/10.1145/1450579.1450627

Digital Library

[44]

Robert C. Zeleznik, Kenneth P. Herndon, and John F. Hughes. 1996. SKETCH: an interface for sketching 3D scenes. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques(SIGGRAPH96). ACM, New York, NY, USA, 163–170. https://doi.org/10.1145/237170.237238

Digital Library

Cited By

Lutnyk LKim KSarbach AKiefer PHäusler RRaubal M(2024)Context-Sensitive Augmented Reality Assistance in the CockpitInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2440976(1-20)Online publication date: 23-Dec-2024
https://doi.org/10.1080/10447318.2024.2440976

Index Terms

3D Flight Planning Using Extended Reality
1. Applied computing
  1. Physical sciences and engineering
    1. Aerospace
2. Human-centered computing
  1. Human computer interaction (HCI)
  2. Visualization
    1. Visualization application domains
      1. Information visualization

Recommendations

Integrated Disruption Management and Flight Planning to Trade Off Delays and Fuel Burn

In this paper we present a novel approach addressing airline delays and recovery. Airline schedule recovery involves making decisions during operations to minimize additional operating costs while getting back on schedule as quickly as possible. The ...
An enhanced rerouting cost estimation algorithm towards internet of drone
Abstract
When a drone flies along with the waypoint flight plan, rerouting has to pay a cost to change its route in real time through the Internet. The inflight rerouting cost estimation algorithm can predict the rerouting cost as a function of the flight ...
Critical Review of Extended Reality Applications in Aviation
Virtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry
Abstract
In recent years, the growing availability of extended reality (XR) technology through mobile, tablet, headset, and wearable devices has led to increased research and development on XR applications in aviation. XR has the potential to enrich the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SUI '24: Proceedings of the 2024 ACM Symposium on Spatial User Interaction

October 2024

396 pages

ISBN:9798400710889

DOI:10.1145/3677386

Editors:
Florian Daiber
DFKI, Saarbrücken, Germany
,
André Zenner
Saarland University, Saarbrücken, Germany
,
Tony Huang
UTS, Sydney, Australia
,
Daniel Zielasko
University of Trier, Trier, Germany
,
Huyen Nguyen
Université Paris-Saclay, Orsay, France
,
Mayra Donaji Barrera Machuca
Dalhousie University, Halifax, Canada

Copyright © 2024 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 October 2024

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SUI '24

Sponsor:

SUI '24: ACM Symposium on Spatial User Interaction

October 7 - 8, 2024

Trier, Germany

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
80
Total Downloads

Downloads (Last 12 months)80
Downloads (Last 6 weeks)6

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lutnyk LKim KSarbach AKiefer PHäusler RRaubal M(2024)Context-Sensitive Augmented Reality Assistance in the CockpitInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2440976(1-20)Online publication date: 23-Dec-2024
https://doi.org/10.1080/10447318.2024.2440976

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten