research-article

Retargeting Technical Documentation to Augmented Reality

Authors:

Bernhard Kerbl,

Michael Donoser,

Dieter Schmalstieg, and

Denis KalkofenAuthors Info & Claims

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

Pages 3337 - 3346

https://doi.org/10.1145/2702123.2702490

Published: 18 April 2015 Publication History

Abstract

We present a system which automatically transfers printed technical documentation, such as handbooks, to three-dimensional Augmented Reality. Our system identifies the most frequent forms of instructions found in printed documentation, such as image sequences, explosion diagrams, textual annotations and arrows indicating motion. The analysis of the printed documentation works automatically, with minimal user input. The system only requires the documentation itself and a CAD model or 3D scan of the object described in the documentation. The output is a fully interactive Augmented Reality application, presenting the information from the printed documentation in 3D, registered to the real object.

Supplementary Material

suppl.mov (pn1936-file3.mp4)

Supplemental video

Download
77.70 MB

MP4 File (p3337-mohr.mp4)

Download
157.86 MB

References

[1]

Agrawala, M., Phan, D., Heiser, J., Haymaker, J., Klingner, J., Hanrahan, P., and Tversky, B. Designing effective step-by-step assembly instructions. ACM Trans. Graph. 22, 3 (2003), 828--837.

Digital Library

[2]

Bergig, O., Hagbi, N., El-Sana, J., and Billinghurst, M. In-place 3d sketching for authoring and augmenting mechanical systems. In Proc. of ISMAR (2009), 87--94.

Digital Library

[3]

Butz, A. Betty: Planning and generating animations for the visualization of movements and spatial relations. In Proc. of Advanced Visual Interfaces (1994), 53--58.

Digital Library

[4]

Canny, J. A computational approach to edge detection. IEEE Trans. Pattern Anal. Mach. Intell. 8, 6 (June 1986), 679--698.

Digital Library

[5]

Dementhon, D. F., and Davis, L. S. Model-based object pose in 25 lines of code. Int. J. Comput. Vision 15, 1--2 (June 1995), 123--141.

Digital Library

[6]

Gupta, A., Fox, D., Curless, B., and Cohen, M. F. Duplotrack a real-time system for authoring and guiding duplo block assembly. In UIST (2012), 389--402.

Digital Library

[7]

Heiser, J., Phan, D., Agrawala, M., Tversky, B., and Hanrahan, P. Identification and validation of cognitive design principles for automated generation of assembly instructions. In Proc. of AVI (2004), 311--319.

Digital Library

[8]

Henderson, S., and Feiner, S. Exploring the benefits of augmented reality documentation for maintenance and repair. IEEE TVCG 17, 10 (2011), 1355--1368.

Digital Library

[9]

Kalkofen, D., Tatzgern, M., and Schmalstieg, D. Explosion diagrams in augmented reality. In Proc of IEEE VR (2009), 71--78.

Digital Library

[10]

Kalkofen, D., Veas, E. E., Zollmann, S., Steinberger, M., and Schmalstieg, D. Adaptive ghosted views for augmented reality. In IEEE ISMAR (2013), 1--9.

[11]

Ledermann, F., and Schmalstieg, D. April: A high-level framework for creating augmented reality presentations. In Proc. of IEEE VR (2005), 187--194.

Digital Library

[12]

Li, W., Agrawala, M., Curless, B., and Salesin, D. Automated generation of interactive 3d exploded view diagrams. ACM Trans. Graph. 27, 3 (2008).

Digital Library

[13]

Li, W., Agrawala, M., and Salesin, D. Interactive image-based exploded view diagrams. In Prof of Graphics Interface (2004), 203--212.

Digital Library

[14]

Liu, C., Yuen, J., Torralba, A., Sivic, J., and Freeman, W. T. Sift flow: Dense correspondence across different scenes. In Proc. of ECCV (2008), 28--42.

Digital Library

[15]

Liu, M.-Y., Tuzel, O., Veeraraghavan, A., and Chellappa, R. Fast directional chamfer matching. In Proc. of CVPR (2010), 1696--1703.

[16]

Lozano-Perez, T. Spatial planning: A configuration space approach. IEEE Trans. Comput. 32, 2 (1983), 108--120.

Digital Library

[17]

Matas, J., Chum, O., Urban, M., and Pajdla, T. Robust wide baseline stereo from maximally stable extremal regions. In Proc. of BMVC (2002).

[18]

Mijksenaar, P., and Westendorp, P. Open Here: The Art of Instructional Design. Thames & Hudson, 1999.

[19]

Newcombe, R. A., Izadi, S., Hilliges, O., Molyneaux, D., Kim, D., Davison, A. J., Kohli, P., Shotton, J., Hodges, S., and Fitzgibbon, A. W. Kinectfusion: Real-time dense surface mapping and tracking. In ISMAR, IEEE (2011), 127--136.

Digital Library

[20]

Petersen, N., and Stricker, D. Learning task structure from video examples for workflow tracking and authoring. In Proc of ISMAR (2012), 237--246.

Digital Library

[21]

Romney, B., Godard, C., Goldwasser, M., and Ramkumar, G. An efficient system for geometric assembly sequence generation and evaluation. In Proc. of Computers in Engineering (1995), 699--712.

[22]

Rusu, R. B., Blodow, N., and Beetz, M. Fast point feature histograms (fpfh) for 3d registration. In ICRA, IEEE (2009), 3212--3217.

Digital Library

[23]

Rusu, R. B., and Cousins, S. 3D is here: Point Cloud Library (PCL). Proc. of ICRA (2011).

[24]

Shao, T., Li, W., Zhou, K., Xu, W., Guo, B., and Mitra, N. J. Interpreting concept sketches. TOG 32, 4 (2013).

Digital Library

[25]

Srinivasan, H., and Gadh, R. Efficient geometric disassembly of multiple components from an assembly using wave propagation. Mech. Design 122 (2000), 179.

[26]

Wilson, R. H., and Latombe, J.-C. Geometric reasoning about mechanical assembly. In Proc. Algorithmic Foundations of Robotics (1995), 203--220.

Digital Library

[27]

Zauner, J., Haller, M., Brandl, A., and Hartmann, W. Authoring of a mixed reality assembly instructor for hierarchical structures. In ISMAR (2003), 237--246.

Digital Library

[28]

Zheng, Y., Kuang, Y., Sugimoto, S., strm, K., and Okutomi, M. Revisiting the pnp problem: A fast, general and optimal solution. In ICCV (2013), 2344--2351.

Digital Library

Cited By

Pohl HHornbæk K(2024)Integrated Calculators: Moving Calculation into the WorldProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661523(343-355)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661523
Borhani ZSharma POrtega F(2024)Survey of Annotations in Extended Reality SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328886930:8(5074-5096)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3288869
Show More Cited By

Index Terms

Retargeting Technical Documentation to Augmented Reality
1. Computing methodologies
  1. Computer graphics
    1. Graphics systems and interfaces
      1. Virtual reality
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
      2. Virtual reality

Recommendations

Retargeting Video Tutorials Showing Tools With Surface Contact to Augmented Reality
CHI '17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems

A video tutorial effectively conveys complex motions, but may be hard to follow precisely because of its restriction to a predetermined viewpoint. Augmented reality (AR) tutorials have been demonstrated to be more effective. We bring the advantages of ...
Read More
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...
Read More
Extending Virtual Reality Display Wall Environments Using Augmented Reality
SUI '19: Symposium on Spatial User Interaction

Two major form factors for virtual reality are head-mounted displays and large display environments such as CAVE®and the LCD-based successor CAVE2®. Each of these has distinct advantages and limitations based on how they’re used. This work explores ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems

April 2015

4290 pages

ISBN:9781450331456

DOI:10.1145/2702123

General Chairs:
Bo Begole
Huawei, USA
,
Jinwoo Kim
Yonsei University, Korea
,
Program Chairs:
Kori Inkpen
Microsoft Research, USA
,
Woontack Woo
KAIST, Korea

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 April 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Competence Centers for Excellent Technologies
Austrian Science Fund

Conference

CHI '15

Sponsor:

SIGCHI

CHI '15: CHI Conference on Human Factors in Computing Systems

April 18 - 23, 2015

Seoul, Republic of Korea

Acceptance Rates

CHI '15 Paper Acceptance Rate 486 of 2,120 submissions, 23%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI PLAY '24

Sponsor:
sigchi

The Annual Symposium on Computer-Human Interaction in Play

October 14 - 17, 2024

Tampere , Finland

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

40
Total Citations
View Citations
1,697
Total Downloads

Downloads (Last 12 months)73
Downloads (Last 6 weeks)7

Other Metrics

View Author Metrics

Citations

Cited By

Pohl HHornbæk K(2024)Integrated Calculators: Moving Calculation into the WorldProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661523(343-355)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661523
Borhani ZSharma POrtega F(2024)Survey of Annotations in Extended Reality SystemsIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.328886930:8(5074-5096)Online publication date: Aug-2024
https://doi.org/10.1109/TVCG.2023.3288869
Becker LNilsson TDemedeiros PRometsch F(2023)Augmented Reality in Service of Human Operations on the Moon: Insights from a Virtual TestbedExtended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems10.1145/3544549.3585860(1-8)Online publication date: 19-Apr-2023
https://dl.acm.org/doi/10.1145/3544549.3585860
Gaarsdal JWolff SMadsen C(2023)Real-Time Exploded View Animation Authoring in VR Based on Simplified Assembly Sequence Planning2023 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW58643.2023.00176(667-668)Online publication date: Mar-2023
https://doi.org/10.1109/VRW58643.2023.00176
Stanescu AMohr PKozinski MMori SSchmalstieg DKalkofen D(2023)State-Aware Configuration Detection for Augmented Reality Step-by-Step Tutorials2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)10.1109/ISMAR59233.2023.00030(157-166)Online publication date: 16-Oct-2023
https://doi.org/10.1109/ISMAR59233.2023.00030
Tainaka KFujimoto YSawabe TKanbara MKato H(2023)Selection framework of visualization methods in designing AR industrial task-support systemsComputers in Industry10.1016/j.compind.2022.103828145(103828)Online publication date: Feb-2023
https://doi.org/10.1016/j.compind.2022.103828
Preda MLavric T(2023)Augmented Reality Training in Manufacturing SectorsThe Digital Twin10.1007/978-3-031-21343-4_17(447-496)Online publication date: 3-Jun-2023
https://doi.org/10.1007/978-3-031-21343-4_17
Hoffmann SPinatti de Carvalho ASchweitzer MAbele NWulf V(2022)Producing and Consuming Instructional Material in Manufacturing Contexts: Evaluation of an AR-based Cyber-Physical Production System for Supporting Knowledge and Expertise SharingProceedings of the ACM on Human-Computer Interaction10.1145/35550916:CSCW2(1-36)Online publication date: 11-Nov-2022
https://dl.acm.org/doi/10.1145/3555091
Skreinig LStanescu AMori SHeyen FMohr PSedlmair MSchmalstieg DKalkofen D(2022)AR Hero: Generating Interactive Augmented Reality Guitar Tutorials2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00086(395-401)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00086
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents