research-article

Heatmaps, Shadows, Bubbles, Rays: Comparing Mid-Air Pen Position Visualizations in Handheld AR

Authors:

Philipp Wacker,

Jan BorchersAuthors Info & Claims

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

Pages 1 - 11

https://doi.org/10.1145/3313831.3376848

Published: 23 April 2020 Publication History

Abstract

In Handheld Augmented Reality, users look at AR scenes through the smartphone held in their hand. In this setting, having a mid-air pointing device like a pen in the other hand greatly expands the interaction possibilities. For example, it lets users create 3D sketches and models while on the go. However, perceptual issues in Handheld AR make it difficult to judge the distance of a virtual object, making it hard to align a pen to it. To address this, we designed and compared different visualizations of the pen's position in its virtual environment, measuring pointing precision, task time, activation patterns, and subjective ratings of helpfulness, confidence, and comprehensibility of each visualization. While all visualizations resulted in only minor differences in precision and task time, subjective ratings of perceived helpfulness and confidence favor a 'heatmap' technique that colors the objects in the scene based on their distance to the pen.

Supplemental Material

MP4 File

Preview video

Download
16.17 MB

MP4 File

Supplemental video

Download
77.89 MB

SRT File

Preview video captions

Download
.56 KB

ZIP File

The supplementary materials provided include the software used in our study, our study recordings, and the scripts used for evaluation. You can find detailed information on how to use the software and scripts in the file ?00Readme.txt?.

Download
47.59 MB

ZIP File

Video figure captions

Download
2.37 KB

References

[1]

Claudia Armbrüster, Marc Wolter, Torsten Kuhlen, Wilhelmus Spijkers, and Bruno Fimm. 2008. Depth Perception in Virtual Reality: Distance Estimations in Peri- and Extrapersonal Space. CyberPsychology & Behavior 11, 1 (2008), 9--15. 18275307.

[2]

Rahul Arora, Rubaiat Habib Kazi, Tovi Grossman, George Fitzmaurice, and Karan Singh. 2018. SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 185, 15 pages.

Digital Library

[3]

Rahul Arora, Rubaiat Habib Kazi, Fraser Anderson, Tovi Grossman, Karan Singh, and George Fitzmaurice. 2017. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17). ACM, NY, NY, USA, 5643--5654.

Digital Library

[4]

Jeff Butterworth, Andrew Davidson, Stephen Hench, and Marc. T. Olano. 1992. 3DM: A Three Dimensional Modeler Using a Head-mounted Display. In Proceedings of the 1992 Symposium on Interactive 3D Graphics (I3D '92). ACM, NY, NY, USA, 135--138.

Digital Library

[5]

James Carpenter and John Bithell. 2000. Bootstrap confidence intervals: when, which, what? A practical guide for medical statisticians. Statistics in Medicine 19, 9 (2000), 1141--1164. 19:91141::AID-SIM4793.0.CO;2-F

[6]

Zeynep Cipiloglu, Abdullah Bulbul, and Tolga Capin. 2010. A Framework for Enhancing Depth Perception in Computer Graphics. In Proceedings of the 7th Symposium on Applied Perception in Graphics and Visualization (APGV '10). ACM, NY, NY, USA, 141--148.

Digital Library

[7]

Geoff Cumming. 2014. The New Statistics: Why and How. Psychological Science 25, 1 (2014), 7--29. 24220629.

[8]

James E. Cutting. 2003. Reconceiving Perceptual Space. In Looking Into Pictures: An interdisciplinary approach to pictorial space, Heiko Hecht, Robert Schwartz, and Margaret Atherton (Eds.). MIT Press, 215--238.

[9]

James E. Cutting and Peter M. Vishton. 1995. Perceiving Layout and Knowing Distances: The Integration, Relative Potency, and Contextual Use of Different Information about Depth. In Perception of Space and Motion, William Epstein and Sheena Rogers (Eds.). Academic Press, San Diego, 69--117.

[10]

Michael F. Deering. 1995. HoloSketch: A Virtual Reality Sketching/Animation Tool. ACM Trans. Comput.-Hum. Interact. 2, 3 (Sept. 1995), 220--238.

Digital Library

[11]

Arindam Dey, Mark Billinghurst, Robert W. Lindeman, and J. Edward Swan. 2018. A Systematic Review of 10 Years of Augmented Reality Usability Studies: 2005 to 2014. Frontiers in Robotics and AI 5 (2018), 37.

[12]

Arindam Dey, Graeme Jarvis, Christian Sandor, and Gerhard Reitmayr. 2012. Tablet versus phone: Depth perception in handheld augmented reality. In 2012 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 187--196.

Digital Library

[13]

Arindam Dey and Christian Sandor. 2014. Lessons Learned: Evaluating Visualizations for Occluded Objects in Handheld Augmented Reality. Int. J. Hum.-Comput. Stud. 72, 10--11 (Oct. 2014), 704--716.

Digital Library

[14]

Catherine Diaz, Michael Walker, Danielle A. Szafir, and Daniel Szafir. 2017. Designing for Depth Perceptions in Augmented Reality. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 111--122.

[15]

Pierre Dragicevic. 2016. Fair Statistical Communication in HCI. Springer International Publishing, 291--330.

[16]

David Drascic and Paul Milgram. 1996. Perceptual issues in augmented reality. In Proc. SPIE Vol. 2653: Stereoscopic Displays and Virtual Reality Systems III. 123--134.

[17]

Bradley Efron. 1987. Better Bootstrap Confidence Intervals. J. Amer. Statist. Assoc. 82, 397 (1987), 171--185.

[18]

Stephen R. Ellis and Brian M. Menges. 1998. Localization of Virtual Objects in the Near Visual Field. Human Factors 40, 3 (1998), 415--431. 11536894.

[19]

Eg Su Goh, Mohd Shahrizal Sunar, and Ajune Wanis Ismail. 2019. 3D Object Manipulation Techniques in Handheld Mobile Augmented Reality Interface: A Review. IEEE Access 7 (2019), 40581--40601.

[20]

Leo Gomba?, Klen ?opi? Pucihar, Matja? Kljun, Paul Coulton, and Jan Grbac. 2016. 3D Virtual Tracing and Depth Perception Problem on Mobile AR. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems (CHI EA '16). ACM, NY, NY, USA, 1849--1856.

Digital Library

[21]

Geoffrey S. Hubona, Philip N. Wheeler, Gregory W. Shirah, and Matthew Brandt. 1999. The Relative Contributions of Stereo, Lighting, and Background Scenes in Promoting 3D Depth Visualization. ACM Trans. Comput.-Hum. Interact. 6, 3 (Sept. 1999), 214--242.

Digital Library

[22]

Ke Huo, Vinayak, and Karthik Ramani. 2017. Window-Shaping: 3D Design Ideation by Creating on, Borrowing from, and Looking at the Physical World. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction (TEI '17). ACM, NY, NY, USA, 37--45.

Digital Library

[23]

Wolfgang Hürst and Casper van Wezel. 2013. Gesture-based interaction via finger tracking for mobile augmented reality. Multimedia Tools and Applications 62, 1 (Jan 2013), 233--258.

[24]

J. Edward Swan II, Liisa Kuparinen, Scott Rapson, and Christian Sandor. 2017. Visually Perceived Distance Judgments: Tablet-Based Augmented Reality Versus the Real World. International Journal of Human-Computer Interaction 33, 7 (2017), 576--591.

[25]

IKEA. 2017. A better reality. Blog. (2017). Retrieved September 17, 2019 from http://highlights.ikea.com/2017/ikea-place/.

[26]

Transparent Statistics in Human-Computer Interaction Working Group. 2019. Transparent Statistics Guidelines. (Jun 2019). (Available at https://transparentstats.github.io/guidelines).

[27]

Bret Jackson and Daniel F. Keefe. 2016. Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR. IEEE Transactions on Visualization and Computer Graphics 22, 4 (April 2016), 1442--1451.

Digital Library

[28]

J. Adam Jones, J. Edward Swan, II, Gurjot Singh, Eric Kolstad, and Stephen R. Ellis. 2008. The Effects of Virtual Reality, Augmented Reality, and Motion Parallax on Egocentric Depth Perception. In Proceedings of the 5th Symposium on Applied Perception in Graphics and Visualization (APGV '08). ACM, NY, NY, USA, 9--14.

Digital Library

[29]

Volkert Jurgens, Andy Cockburn, and Mark Billinghurst. 2006. Depth Cues for Augmented Reality Stakeout. In Proceedings of the 7th ACM SIGCHI New Zealand Chapter's International Conference on Computer-human Interaction: Design Centered HCI (CHINZ '06). ACM, NY, NY, USA, 117--124.

Digital Library

[30]

Ernst Kruijff, J. Edward Swan, and Steven Feiner. 2010. Perceptual issues in augmented reality revisited. In 2010 IEEE International Symposium on Mixed and Augmented Reality. 3--12.

[31]

David Lakatos, Matthew Blackshaw, Alex Olwal, Zachary Barryte, Ken Perlin, and Hiroshi Ishii. 2014. T(Ether): Spatially-aware Handhelds, Gestures and Proprioception for Multi-user 3D Modeling and Animation. In Proceedings of the 2Nd ACM Symposium on Spatial User Interaction (SUI '14). ACM, NY, NY, USA, 90--93.

Digital Library

[32]

Ingmar Lissner and Philipp Urban. 2012. Toward a Unified Color Space for Perception-Based Image Processing. IEEE Transactions on Image Processing 21, 3 (March 2012), 1153--1168.

Digital Library

[33]

Mark A. Livingston, Zhuming Ai, J. Edward Swan, and Harvey S. Smallman. 2009. Indoor vs. Outdoor Depth Perception for Mobile Augmented Reality. In 2009 IEEE Virtual Reality Conference. 55--62.

Digital Library

[34]

Paul Milgram and Fumio Kishino. 1994. A Taxonomy of Mixed Reality Visual Displays. IEICE Transactions on Information Systems E77-D (1994). http://vered. rose.utoronto.ca/people/paul_dir/IEICE94/ieice.html

[35]

Annette Mossel, Benjamin Venditti, and Hannes Kaufmann. 2013. 3DTouch and HOMER-S: Intuitive Manipulation Techniques for One-handed Handheld Augmented Reality. In Proceedings of the Virtual Reality International Conference: Laval Virtual (VRIC '13). ACM, NY, NY, USA, Article 12, 10 pages.

Digital Library

[36]

Huaishu Peng, Jimmy Briggs, Cheng-Yao Wang, Kevin Guo, Joseph Kider, Stefanie Mueller, Patrick Baudisch, and François Guimbretière. 2018. RoMA: Interactive Fabrication with Augmented Reality and a Robotic 3D Printer. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, NY, NY, USA, Article 579, 12 pages.

Digital Library

[37]

Nina Rosa, Remco C. Veltkamp, Wolfgang Hürst, Tanja Nijboer, Carolien Gilbers, and Peter Werkhoven. 2019. The Supernumerary Hand Illusion in Augmented Reality. ACM Trans. Appl. Percept. 16, 2, Article 12 (Aug. 2019), 20 pages.

Digital Library

[38]

Emanuel Sachs, Andrew Roberts, and David Stoops. 1991. 3-Draw: a tool for designing 3D shapes. IEEE Computer Graphics and Applications 11, 6 (Nov 1991), 18--26.

Digital Library

[39]

Valentin Schwind, Pascal Knierim, Lewis Chuang, and Niels Henze. 2017. "Where's Pinky?": The Effects of a Reduced Number of Fingers in Virtual Reality. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (CHI PLAY '17). ACM, NY, NY, USA, 507--515.

Digital Library

[40]

Iris Seidinger and Jens Grubert. 2016. 3D Character Customization for Non-Professional Users in Handheld Augmented Reality. In 2016 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 129--134.

[41]

J. Edward Swan, Mark A. Livingston, Harvey S. Smallman, Dennis Brown, Yohan Baillot, Joseph L. Gabbard, and Deborah Hix. 2006. A Perceptual Matching Technique for Depth Judgments in Optical, See-Through Augmented Reality. In IEEE Virtual Reality Conference (VR 2006). 19--26.

Digital Library

[42]

Jeff K. T. Tang, Tin-Young A. Duong, Yui-Wang Ng, and Hoi-Kit Luk. 2015. Learning to create 3D models via an augmented reality smartphone interface. In 2015 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE). 236--241.

[43]

Takahiro Tsuda, Haruyoshi Yamamoto, Yoshinari Kameda, and Yuichi Ohta. 2005. Visualization Methods for Outdoor See-through Vision. In Proceedings of the 2005 International Conference on Augmented Tele-existence (ICAT '05). ACM, NY, NY, USA, 62--69.

Digital Library

[44]

Klen Copic Pucihar, Paul Coulton, and Jason Alexander. 2013. Evaluating Dual-view Perceptual Issues in Handheld Augmented Reality: Device vs. User Perspective Rendering. In Proceedings of the 15th ACM on International Conference on Multimodal Interaction (ICMI '13). ACM, NY, NY, USA, 381--388.

Digital Library

[45]

Philipp Wacker, Oliver Nowak, Simon Voelker, and Jan Borchers. 2019. ARPen: Mid-Air Object Manipulation Techniques for a Bimanual AR System with Pen & Smartphone. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI '19). ACM, NY, NY, USA, Article 619, 12 pages.

Digital Library

[46]

Philipp Wacker, Adrian Wagner, Simon Voelker, and Jan Borchers. 2018. Physical Guides: An Analysis of 3D Sketching Performance on Physical Objects in Augmented Reality. In Proceedings of the Symposium on Spatial User Interaction (SUI '18). ACM, NY, NY, USA, 25--35.

Digital Library

[47]

Leonard R. Wanger, James A. Ferwerda, and Donald P. Greenberg. 1992. Perceiving spatial relationships in computer-generated images. IEEE Computer Graphics and Applications 12, 3 (May 1992), 44--58.

Digital Library

[48]

John P. Wann, Simon Rushton, and Mark Mon-Williams. 1995. Natural problems for stereoscopic depth perception in virtual environments. Vision Research 35, 19 (1995), 2731--2736.

[49]

Jason Wither and Tobias Hollerer. 2005. Pictorial depth cues for outdoor augmented reality. In Ninth IEEE International Symposium on Wearable Computers (ISWC'05). 92--99.

Digital Library

[50]

Po-Chen Wu, Robert Wang, Kenrick Kin, Christopher Twigg, Shangchen Han, Ming-Hsuan Yang, and Shao-Yi Chien. 2017. DodecaPen: Accurate 6DoF Tracking of a Passive Stylus. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology (UIST '17). ACM, NY, NY, USA, 365--374.

Digital Library

[51]

Min Xin, Ehud Sharlin, and Mario Costa Sousa. 2008. Napkin sketch: handheld mixed reality 3D sketching. In Proceedings of the 2008 ACM symposium on Virtual reality software and technology - VRST '08. ACM Press.

Digital Library

Cited By

Takeyama YNagayama KChoi MShizuki B(2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682078
Zawish MDharejo FKhowaja SRaza SDavy SDev KBellavista P(2024)AI and 6G Into the Metaverse: Fundamentals, Challenges and Future Research TrendsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.33494655(730-778)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3349465
Kamińska DZwoliński GLaska-Leśniewicz ARaposo RVairinhos MPereira EUrem FLjubić Hinić MHaamer RAnbarjafari G(2023)Augmented Reality: Current and New Trends in EducationElectronics10.3390/electronics1216353112:16(3531)Online publication date: 21-Aug-2023
https://doi.org/10.3390/electronics12163531
Show More Cited By

Index Terms

Heatmaps, Shadows, Bubbles, Rays: Comparing Mid-Air Pen Position Visualizations in Handheld AR
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Mixed / augmented reality
    2. Interaction techniques

Recommendations

Evaluating Menu Techniques for Handheld AR with a Smartphone & Mid-Air Pen
MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

Adding a mid-air pen to Handheld Augmented Reality creates a new kind of bimanual interaction for which many fundamental interaction design questions have not been answered yet. In particular, menus are an essential component in most visual interfaces, ...
3D Tabletop AR: A Comparison of Mid-Air, Touch and Touch+Mid-Air Interaction
AVI '20: Proceedings of the 2020 International Conference on Advanced Visual Interfaces

This paper contributes a first comparative study of three techniques for selecting 3D objects anchored to the table in tabletop Augmented Reality (AR). The impetus for this study is that touch interaction makes more sense when the targeted objects are ...
Head-mounted display with mid-air tactile feedback
VRST '15: Proceedings of the 21st ACM Symposium on Virtual Reality Software and Technology

Virtual and physical worlds are merging. Currently users of head-mounted displays cannot have unobtrusive tactile feedback while touching virtual objects. We present a mid-air tactile feedback system for head-mounted displays. Our prototype uses the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems

April 2020

10688 pages

ISBN:9781450367080

DOI:10.1145/3313831

General Chairs:
Regina Bernhaupt
Eindhoven University of Technology, Netherlands
,
Florian 'Floyd' Mueller
Monash University, Australia
,
David Verweij
Newcastle University, UK
,
Josh Andres
RMIT, Australia
,
Program Chairs:
Joanna McGrenere
University of British Columbia, Canada
,
Andy Cockburn
University of Canterbury, New Zealand
,
Ignacio Avellino
University of Maryland Baltimore County, USA
,
Alix Goguey
Grenoble Alpes University, France
,
Pernille Bjørn
University of Copenhagen, Denmark
,
Shengdong (Shen) Zhao
National University of Singapore, Singapore
,
Briane Paul Samson
Future University Hakodate, Japan & De La Salle University, Philippines
,
Rafal Kocielnik
University of Washington, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '20

Sponsor:

SIGCHI

CHI '20: CHI Conference on Human Factors in Computing Systems

April 25 - 30, 2020

HI, Honolulu, USA

Acceptance Rates

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

11
Total Citations
View Citations
691
Total Downloads

Downloads (Last 12 months)51
Downloads (Last 6 weeks)2

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Takeyama YNagayama KChoi MShizuki B(2024)PhoneCanvas: 3D Sketching System Using a Depth Camera-Equipped Smartphone as a CanvasProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682078(1-12)Online publication date: 7-Oct-2024
https://dl.acm.org/doi/10.1145/3677386.3682078
Zawish MDharejo FKhowaja SRaza SDavy SDev KBellavista P(2024)AI and 6G Into the Metaverse: Fundamentals, Challenges and Future Research TrendsIEEE Open Journal of the Communications Society10.1109/OJCOMS.2024.33494655(730-778)Online publication date: 2024
https://doi.org/10.1109/OJCOMS.2024.3349465
Kamińska DZwoliński GLaska-Leśniewicz ARaposo RVairinhos MPereira EUrem FLjubić Hinić MHaamer RAnbarjafari G(2023)Augmented Reality: Current and New Trends in EducationElectronics10.3390/electronics1216353112:16(3531)Online publication date: 21-Aug-2023
https://doi.org/10.3390/electronics12163531
Hueber SWilhelm JSchäfer RVoelker SBorchers J(2023)User-Aware Rendering: Merging the Strengths of Device- and User-Perspective Rendering in Handheld ARProceedings of the ACM on Human-Computer Interaction10.1145/36042787:MHCI(1-18)Online publication date: 13-Sep-2023
https://dl.acm.org/doi/10.1145/3604278
Zhou YYang PXu XShao BFeng GLiu JLuo B(2023)FMASketch: Freehand Mid-Air Sketching in ARInternational Journal of Human–Computer Interaction10.1080/10447318.2023.222394840:8(2142-2152)Online publication date: 22-Jun-2023
https://doi.org/10.1080/10447318.2023.2223948
Venugopal JSubramanian APeatchimuthu J(2023)The realm of metaverse: A surveyComputer Animation and Virtual Worlds10.1002/cav.215034:5Online publication date: 2-Mar-2023
https://doi.org/10.1002/cav.2150
Babic TReiterer HHaller M(2022)Understanding and Creating Spatial Interactions with Distant Displays Enabled by Unmodified Off-The-Shelf SmartphonesMultimodal Technologies and Interaction10.3390/mti61000946:10(94)Online publication date: 19-Oct-2022
https://doi.org/10.3390/mti6100094
Lee LBraud THosio SHui P(2021)Towards Augmented Reality Driven Human-City Interaction: Current Research on Mobile Headsets and Future ChallengesACM Computing Surveys10.1145/346796354:8(1-38)Online publication date: 4-Oct-2021
https://dl.acm.org/doi/10.1145/3467963
Qian JZhou TYoung-Ng MMa JCheung ALi XGonsher IHuang J(2021)Portalware: Exploring Free-Hand AR Drawing with a Dual-Display Smartphone-Wearable ParadigmProceedings of the 2021 ACM Designing Interactive Systems Conference10.1145/3461778.3462098(205-219)Online publication date: 28-Jun-2021
https://dl.acm.org/doi/10.1145/3461778.3462098
Wacker PNowak OVoelker SBorchers J(2020)Evaluating Menu Techniques for Handheld AR with a Smartphone & Mid-Air Pen22nd International Conference on Human-Computer Interaction with Mobile Devices and Services10.1145/3379503.3403548(1-10)Online publication date: 5-Oct-2020
https://dl.acm.org/doi/10.1145/3379503.3403548
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten