research-article

TutorialLens: Authoring Interactive Augmented Reality Tutorials Through Narration and Demonstration

Authors:

Jeffrey P Bigham,

Anhong GuoAuthors Info & Claims

SUI '21: Proceedings of the 2021 ACM Symposium on Spatial User Interaction

Article No.: 16, Pages 1 - 11

https://doi.org/10.1145/3485279.3485289

Published: 09 November 2021 Publication History

Abstract

Exploring unfamiliar devices and interfaces through trial and error can be challenging and frustrating. Existing video tutorials require frequent context switching between the device showing the tutorial and the device being used. While augmented reality (AR) has been adopted to create user manuals, many are inflexible for diverse tasks, and usually require programming and AR development experience. We present TutorialLens, a system for authoring interactive AR tutorials through narration and demonstration. To use TutorialLens, authors demonstrate tasks step-by-step while verbally explaining what they are doing. TutorialLens automatically detects and records 3D finger positions and guides authors to capture important changes of the device. Using the created tutorials, TutorialLens then provides AR visual guidance and feedback for novice device users to complete the demonstrated tasks. TutorialLens is automated, friendly to users without AR development experience, and applicable to a variety of devices and tasks.

Supplementary Material

Supplemental files (3485279.3485289.zip)

Download
688.23 KB

MP4 File (video_figure.mp4)

Supplemental video

Download
528.98 MB

References

[1]

Nagore Barrena, Andrés Navarro, Sara García, and David Oyarzun. 2016. CoolTour: VR and AR authoring tool to create cultural experiences. In Intelligent Interactive Multimedia Systems and Services 2016. Springer, 483–489.

[2]

Martin Bauer, Bernd Bruegge, Gudrun Klinker, Asa MacWilliams, Thomas Reicher, Stefan Riss, Christian Sandor, and Martin Wagner. 2001. Design of a component-based augmented reality framework. In Proceedings IEEE and ACM International Symposium on Augmented Reality. IEEE, 45–54.

[3]

Jonas Blattgerste, Benjamin Strenge, Patrick Renner, Thies Pfeiffer, and Kai Essig. 2017. Comparing conventional and augmented reality instructions for manual assembly tasks. In Proceedings of the 10th International Conference on Pervasive Technologies related to Assistive Environments. 75–82.

Digital Library

[4]

G. Bradski. 2000. The OpenCV Library. Dr. Dobb’s Journal of Software Tools(2000).

[5]

Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. 2018. OpenPose: realtime multi-person 2D pose estimation using Part Affinity Fields. arXiv preprint arXiv:1812.08008(2018).

[6]

Pei-Yu Chi, Sally Ahn, Amanda Ren, Mira Dontcheva, Wilmot Li, and Björn Hartmann. 2012. MixT: automatic generation of step-by-step mixed media tutorials. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. 93–102.

Digital Library

[7]

Pei-Yu Chi, Joyce Liu, Jason Linder, Mira Dontcheva, Wilmot Li, and Bjoern Hartmann. 2013. Democut: generating concise instructional videos for physical demonstrations. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology. 141–150.

Digital Library

[8]

Subramanian Chidambaram, Hank Huang, Fengming He, Xun Qian, Ana M Villanueva, Thomas S Redick, Wolfgang Stuerzlinger, and Karthik Ramani. 2021. ProcessAR: An Augmented Reality-Based Tool to Create In-Situ Procedural 2D/3D AR Instructions. In Proceedings of the Designing Interactive Systems Conference.

Digital Library

[9]

Tom Djajadiningrat, Pei-Yin Chao, SeYoung Kim, Marleen Van Leengoed, and Jeroen Raijmakers. 2016. Mime: An AR-based System Helping Patients to Test their Blood at Home. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. 347–359.

Digital Library

[10]

Tom Djajadiningrat, Patray Lui, Pei-Yin Chao, and Christian Richard. 2016. Virtual Trainer: A Low Cost AR Simulation of a Sudden Cardiac Arrest Emergency. In Proceedings of the 2016 ACM Conference on Designing Interactive Systems. 607–618.

Digital Library

[11]

Ralf Dörner, Christian Geiger, Michael Haller, and Volker Paelke. 2003. Authoring mixed reality—a component and framework-based approach. In Entertainment Computing. Springer, 405–413.

[12]

Markus Funk, Thomas Kosch, and Albrecht Schmidt. 2016. Interactive worker assistance: comparing the effects of in-situ projection, head-mounted displays, tablet, and paper instructions. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing. 934–939.

Digital Library

[13]

Jesús Gimeno, Pedro Morillo, Juan Manuel Orduña, and Marcos Fernández. 2013. An easy-to-use ar authoring tool for industrial applications. In Computer Vision, Imaging and Computer Graphics. Theory and Application. Springer, 17–32.

[14]

Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2010. Chronicle: capture, exploration, and playback of document workflow histories. In Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology. 143–152.

Digital Library

[15]

Anhong Guo, Junhan Kong, Michael Rivera, Frank F Xu, and Jeffrey P Bigham. 2019. StateLens: A Reverse Engineering Solution for Making Existing Dynamic Touchscreens Accessible. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 371–385.

Digital Library

[16]

Ankit Gupta, Dieter Fox, Brian Curless, and Michael Cohen. 2012. DuploTrack: a real-time system for authoring and guiding duplo block assembly. In Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. 389–402.

Digital Library

[17]

Pavel Gurevich, Joel Lanir, Benjamin Cohen, and Ran Stone. 2012. TeleAdvisor: a versatile augmented reality tool for remote assistance. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems. 619–622.

Digital Library

[18]

Steven Henderson and Steven Feiner. 2010. Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE Transactions on Visualization and Computer Graphics 17, 10(2010), 1355–1368.

Digital Library

[19]

Steven J Henderson and Steven K Feiner. 2011. Augmented reality in the psychomotor phase of a procedural task. In 2011 10th IEEE International Symposium on Mixed and Augmented Reality. IEEE, 191–200.

Digital Library

[20]

Junhan Kong, Anhong Guo, and Jeffrey P Bigham. 2019. Supporting Older Adults in Using Complex User Interfaces with Augmented Reality. In The 21st International ACM SIGACCESS Conference on Computers and Accessibility. 661–663.

[21]

Taehee Lee and Tobias Hollerer. 2008. Hybrid feature tracking and user interaction for markerless augmented reality. In 2008 IEEE Virtual Reality Conference. IEEE, 145–152.

[22]

Rock Leung, Charlotte Tang, Shathel Haddad, Joanna Mcgrenere, Peter Graf, and Vilia Ingriany. 2012. How older adults learn to use mobile devices: Survey and field investigations. ACM Transactions on Accessible Computing (TACCESS) 4, 3 (2012), 1–33.

Digital Library

[23]

Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, 2019. MediaPipe: A Framework for Building Perception Pipelines. arXiv preprint arXiv:1906.08172(2019).

[24]

Blair MacIntyre, Maribeth Gandy, Steven Dow, and Jay David Bolter. 2004. DART: a toolkit for rapid design exploration of augmented reality experiences. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology. 197–206.

Digital Library

[25]

Nikoletta Mavrogeorgi, Stefanos Koutsoutos, Angelos Yannopoulos, Theodora Varvarigou, and George Kambourakis. 2009. Cultural heritage experience with virtual reality according to user preferences. In 2009 Second International Conference on Advances in Human-Oriented and Personalized Mechanisms, Technologies, and Services. IEEE, 13–18.

Digital Library

[26]

Peter Mohr, David Mandl, Markus Tatzgern, Eduardo Veas, Dieter Schmalstieg, and Denis Kalkofen. 2017. Retargeting video tutorials showing tools with surface contact to augmented reality. In Proceedings of the 2017 ACM CHI Conference on Human Factors in Computing Systems. 6547–6558.

Digital Library

[27]

Tam V Nguyen, Bilal Mirza, Dorothy Tan, and Jose Sepulveda. 2018. ASMIM: Augmented Reality Authoring System for Mobile Interactive Manuals. In Proceedings of the 12th International Conference on Ubiquitous Information Management and Communication. 1–6.

Digital Library

[28]

Alexander Plopski, Varunyu Fuvattanasilp, Jarkko Poldi, Takafumi Taketomi, Christian Sandor, and Hirokazu Kato. 2018. Efficient in-situ creation of augmented reality tutorials. In 2018 Workshop on Metrology for Industry 4.0 and IoT. IEEE, 7–11.

[29]

Suporn Pongnumkul, Mira Dontcheva, Wilmot Li, Jue Wang, Lubomir Bourdev, Shai Avidan, and Michael F Cohen. 2011. Pause-and-play: automatically linking screencast video tutorials with applications. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology. 135–144.

Digital Library

[30]

Dariusz Rumiński and Krzysztof Walczak. 2013. Creation of interactive ar content on mobile devices. In International Conference on Business Information Systems. Springer, 258–269.

[31]

Hartmut Seichter, Julian Looser, and Mark Billinghurst. 2008. ComposAR: An intuitive tool for authoring AR applications. In 2008 7th IEEE/ACM International Symposium on Mixed and Augmented Reality. IEEE, 177–178.

Digital Library

[32]

Anna Syberfeldt, Oscar Danielsson, Magnus Holm, and Lihui Wang. 2015. Visual assembling guidance using augmented reality. Procedia Manufacturing 1(2015), 98–109.

[33]

Keishi Tainaka, Yuichiro Fujimoto, Masayuki Kanbara, Hirokazu Kato, Atsunori Moteki, Kensuke Kuraki, Kazuki Osamura, Toshiyuki Yoshitake, and Toshiyuki Fukuoka. 2020. Guideline and Tool for Designing an Assembly Task Support System Using Augmented Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 486–497.

[34]

Anh Truong, Floraine Berthouzoz, Wilmot Li, and Maneesh Agrawala. 2016. Quickcut: An interactive tool for editing narrated video. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 497–507.

Digital Library

[35]

Zeyu Wang, Shiyu Qiu, Qingyang Chen, Natallia Trayan, Alexander Ringlein, Julie Dorsey, and Holly Rushmeier. 2019. AniCode: authoring coded artifacts for network-free personalized animations. The Visual Computer 35, 6 (2019), 885–897.

Digital Library

[36]

Matt Whitlock, George Fitzmaurice, Tovi Grossman, and Justin Matejka. 2019. AuthAR: Concurrent Authoring of Tutorials for AR Assembly Guidance. (2019).

Cited By

Lipkowitz G(2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688875
Li JKristensson P(2024)On the Benefits of Image-Schematic Metaphors when Designing Mixed Reality SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642925(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642925
Bock LBohné TTadeja S(2024)Decision support for augmented reality-based assistance systems deployment in industrial settingsMultimedia Tools and Applications10.1007/s11042-024-19861-xOnline publication date: 13-Aug-2024
https://doi.org/10.1007/s11042-024-19861-x
Show More Cited By

Recommendations

GesturAR: An Authoring System for Creating Freehand Interactive Augmented Reality Applications
UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Freehand gesture is an essential input modality for modern Augmented Reality (AR) user experiences. However, developing AR applications with customized hand interactions remains a challenge for end-users. Therefore, we propose GesturAR, an end-to-end ...
Designing and implementing interactive and realistic augmented reality experiences

In this paper, we propose an approach for supporting the design and implementation of interactive and realistic Augmented Reality (AR). Despite the advances in AR technology, most software applications still fail to support AR experiences where virtual ...
Interactive Augmented Live Virtual Reality Streaming: A Health Care Application
ICMHI '18: Proceedings of the 2nd International Conference on Medical and Health Informatics

Virtual Reality (VR) technology has been around for decades; however, we have probably only begun to realize the practical applications until recent years. With increase of computing power, decrease of cost and physical dimension, a new class of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SUI '21: Proceedings of the 2021 ACM Symposium on Spatial User Interaction

November 2021

206 pages

ISBN:9781450390910

DOI:10.1145/3485279

Editors:
Francisco Ortega
Colorado State University, USA
,
Robert Teather
Carleton University, Canada
,
Gerd Bruder
University of Central Florida, USA
,
Thammathip Piumsomboon
University of Canterbury, New Zealand
,
Benjamin Weyers
University of Trier, Germany
,
Anil Ufuk Batmaz
Kadir Has University, Turkey
,
Kyle Johnsen
University of Georgia, USA
,
Christoph Borst
University of Louisiana at Lafayette, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 09 November 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

SUI '21

Sponsor:

SUI '21: Symposium on Spatial User Interaction

November 9 - 10, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 86 of 279 submissions, 31%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
339
Total Downloads

Downloads (Last 12 months)94
Downloads (Last 6 weeks)6

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Lipkowitz G(2024)RubiXR: Demonstration of dynamic task augmentation through co-design of interactive 3D content and 3D user interfacesProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3688875(1-3)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3688875
Li JKristensson P(2024)On the Benefits of Image-Schematic Metaphors when Designing Mixed Reality SystemsProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642925(1-20)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642925
Bock LBohné TTadeja S(2024)Decision support for augmented reality-based assistance systems deployment in industrial settingsMultimedia Tools and Applications10.1007/s11042-024-19861-xOnline publication date: 13-Aug-2024
https://doi.org/10.1007/s11042-024-19861-x
Turakhia DJiang PMueller S(2023)The Reflective Make-AR In-Action: Using Augmented Reality for Reflection-based Learning of MakerskillsExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585850(1-6)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585850
Liu ZZhu ZJiang EHuang FVillanueva AQian XWang TRamani K(2023)InstruMentAR: Auto-Generation of Augmented Reality Tutorials for Operating Digital Instruments Through Recording Embodied DemonstrationProceedings of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544548.3581442(1-17)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544548.3581442
Ogawa K(2022)Proposal for Display System Superimposing Operating Instructions on Home Appliances2022 IEEE 4th Global Conference on Life Sciences and Technologies (LifeTech)10.1109/LifeTech53646.2022.9754963(78-80)Online publication date: 7-Mar-2022
https://doi.org/10.1109/LifeTech53646.2022.9754963
Tobisková NMalmsköld LPederson T(2022)Multimodal Augmented Reality and Subtle Guidance for Industrial Assembly – A Survey and Ideation MethodVirtual, Augmented and Mixed Reality: Applications in Education, Aviation and Industry10.1007/978-3-031-06015-1_23(329-349)Online publication date: 26-Jun-2022
https://dl.acm.org/doi/10.1007/978-3-031-06015-1_23
Kosch TKnierim PKritzler MBeicht DMichahelles F(2022)Lenssembly: Authoring Assembly Instructions in Augmented Reality Using Programming-by-DemonstrationHuman-Technology Interaction10.1007/978-3-030-99235-4_8(199-222)Online publication date: 14-Dec-2022
https://doi.org/10.1007/978-3-030-99235-4_8

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents