research-article

AR-enhanced Widgets for Smartphone-centric Interaction

Authors:

Eugenie Brasier,

Emmanuel Pietriga,

Caroline AppertAuthors Info & Claims

MobileHCI '21: Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction

Article No.: 32, Pages 1 - 12

https://doi.org/10.1145/3447526.3472019

Published: 27 September 2021 Publication History

Abstract

We contribute a detailed investigation of AR-enhanced widgets for smartphones, where AR technology is not only used to offload widgets from the phone to the air around it, but to give users more control on input precision as well. Such widgets have the obvious benefit of freeing up screen real-estate on the phone, but their other potential benefits remain largely theoretical. Their limitations are not well understood, most particularly in terms of input performance. We compare different AR-enhanced widget designs against their state-of-the-art touch-only counterparts with a series of exploratory studies in which participants had to perform three tasks: command trigger, parameter value adjustment, and precise 2D selection. We then derive guidelines from our empirical observations.

References

[1]

David Ahlström, Khalad Hasan, and Pourang Irani. 2014. Are You Comfortable Doing That? Acceptance Studies of around-Device Gestures in and for Public Settings. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices and Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 193–202. https://doi.org/10.1145/2628363.2628381

Digital Library

[2]

Caroline Appert, Olivier Chapuis, and Emmanuel Pietriga. 2010. High-Precision Magnification Lenses. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI ’10). Association for Computing Machinery, New York, NY, USA, 273–282. https://doi.org/10.1145/1753326.1753366

Digital Library

[3]

Caroline Appert and Jean-Daniel Fekete. 2006. OrthoZoom Scroller: 1D Multi-Scale Navigation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 21–30. https://doi.org/10.1145/1124772.1124776

Digital Library

[4]

Daniel Ashbrook, Patrick Baudisch, and Sean White. 2011. Nenya: Subtle and Eyes-Free Mobile Input with a Magnetically-Tracked Finger Ring. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI ’11). Association for Computing Machinery, New York, NY, USA, 2043–2046. https://doi.org/10.1145/1978942.1979238

Digital Library

[5]

Eugenie Brasier, Olivier Chapuis, Nicolas Ferey, Jeanne Vezien, and Caroline Appert. 2020. ARPads: Mid-air Indirect Input for Augmented Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 332–343. https://doi.org/10.1109/ISMAR50242.2020.00060

[6]

Frederik Brudy, Christian Holz, Roman Rädle, Chi-Jui Wu, Steven Houben, Clemens Nylandsted Klokmose, and Nicolai Marquardt. 2019. Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning Across Multiple Devices. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–28. https://doi.org/10.1145/3290605.3300792

Digital Library

[7]

Rahul Budhiraja, Gun. A. Lee, and Mark Billinghurst. 2013. Using a HHD with a HMD for mobile AR interaction. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 1–6. https://doi.org/10.1109/ISMAR.2013.6671837

[8]

Alex Butler, Shahram Izadi, and Steve Hodges. 2008. SideSight: Multi-”touch” Interaction around Small Devices. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology (Monterey, CA, USA) (UIST ’08). Association for Computing Machinery, New York, NY, USA, 201–204. https://doi.org/10.1145/1449715.1449746

Digital Library

[9]

William Buxton. 1990. A Three-State Model of Graphical Input. In Proceedings of the IFIP TC13 Third Interational Conference on Human-Computer Interaction(INTERACT ’90). North-Holland Publishing Co., NLD, 449–456.

Digital Library

[10]

Xiang “Anthony” Chen, Nicolai Marquardt, Anthony Tang, Sebastian Boring, and Saul Greenberg. 2012. Extending a Mobile Device’s Interaction Space through Body-Centric Interaction. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (San Francisco, California, USA) (MobileHCI ’12). Association for Computing Machinery, New York, NY, USA, 151–160. https://doi.org/10.1145/2371574.2371599

Digital Library

[11]

Xiang “Anthony” Chen, Julia Schwarz, Chris Harrison, Jennifer Mankoff, and Scott E. Hudson. 2014. Air+touch: Interweaving Touch & in-Air Gestures. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 519–525. https://doi.org/10.1145/2642918.2647392

Digital Library

[12]

John Dudley, Hrovje Benko, Daniel Wigdor, and Per Ola Kristensson. 2019. Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality. In Proceedings of the International Symposium on Mixed and Augmented Reality(ISMAR ’19). IEEE, 289–300. https://doi.org/10.1109/ISMAR.2019.00027

[13]

Anna Eiberger, Per Ola Kristensson, Susanne Mayr, Matthias Kranz, and Jens Grubert. 2019. Effects of Depth Layer Switching between an Optical See-Through Head-Mounted Display and a Body-Proximate Display. In Symposium on Spatial User Interaction (New Orleans, LA, USA) (SUI ’19). Association for Computing Machinery, New York, NY, USA, Article 15, 9 pages. https://doi.org/10.1145/3357251.3357588

Digital Library

[14]

Tovi Grossman, Ken Hinckley, Patrick Baudisch, Maneesh Agrawala, and Ravin Balakrishnan. 2006. Hover Widgets: Using the Tracking State to Extend the Capabilities of Pen-Operated Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Montréal, Québec, Canada) (CHI ’06). Association for Computing Machinery, New York, NY, USA, 861–870. https://doi.org/10.1145/1124772.1124898

Digital Library

[15]

Jens Grubert, Matthias Heinisch, Aaron Quigley, and Dieter Schmalstieg. 2015. MultiFi: Multi Fidelity Interaction with Displays On and Around the Body. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 3933–3942. https://doi.org/10.1145/2702123.2702331

Digital Library

[16]

Jens Grubert and Matthias Kranz. 2017. HeadPhones: Ad Hoc Mobile Multi-Display Environments through Head Tracking. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 3966–3971. https://doi.org/10.1145/3025453.3025533

Digital Library

[17]

Jens Grubert, Matthias Kranz, and Aaron Quigley. 2016. Challenges in mobile multi-device ecosystems. mUX: The Journal of Mobile User Experience 5, 1 (2016), 5. https://doi.org/10.1186/s13678-016-0007-y

[18]

Jens Grubert, Eyal Ofek, Michel Pahud, Matthias Kranz, and Dieter Schmalstieg. 2016. GlassHands: Interaction Around Unmodified Mobile Devices Using Sunglasses. In Proceedings of the ACM International Conference on Interactive Surfaces and Spaces (Niagara Falls, Ontario, Canada) (ISS ’16). Association for Computing Machinery, New York, NY, USA, 215–224. https://doi.org/10.1145/2992154.2992162

Digital Library

[19]

Jaehyun Han, Sunggeun Ahn, Keunwoo Park, and Geehyuk Lee. 2017. Designing Touch Gestures Using the Space around the Smartwatch as Continuous Input Space. In Proceedings of the ACM International Conference on Interactive Surfaces and Spaces (Brighton, United Kingdom) (ISS ’17). Association for Computing Machinery, New York, NY, USA, 210–219. https://doi.org/10.1145/3132272.3134134

Digital Library

[20]

Chris Harrison and Scott E. Hudson. 2009. Abracadabra: Wireless, High-Precision, and Unpowered Finger Input for Very Small Mobile Devices. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology (Victoria, BC, Canada) (UIST ’09). Association for Computing Machinery, New York, NY, USA, 121–124. https://doi.org/10.1145/1622176.1622199

Digital Library

[21]

Khalad Hasan, David Ahlström, Junhyeok Kim, and Pourang Irani. 2017. AirPanes: Two-Handed Around-Device Interaction for Pane Switching on Smartphones. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 679–691. https://doi.org/10.1145/3025453.3026029

Digital Library

[22]

Juan David Hincapié-Ramos, Xiang Guo, Paymahn Moghadasian, and Pourang Irani. 2014. Consumed Endurance: A Metric to Quantify Arm Fatigue of Mid-Air Interactions. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI ’14). Association for Computing Machinery, New York, NY, USA, 1063–1072. https://doi.org/10.1145/2556288.2557130

Digital Library

[23]

ISO. 2000. 9241-9 Ergonomic requirements for office work with visual display terminals (VDTs)-Part 9: Requirements for non-keyboard input devices. International Organization for Standardization (2000).

[24]

Brett Jones, Rajinder Sodhi, David Forsyth, Brian Bailey, and Giuliano Maciocci. 2012. Around Device Interaction for Multiscale Navigation. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (San Francisco, California, USA) (MobileHCI ’12). Association for Computing Machinery, New York, NY, USA, 83–92. https://doi.org/10.1145/2371574.2371589

Digital Library

[25]

Hamed Ketabdar, Mehran Roshandel, and Kamer Ali Yüksel. 2010. Towards Using Embedded Magnetic Field Sensor for around Mobile Device 3D Interaction. In Proceedings of the 12th International Conference on Human Computer Interaction with Mobile Devices and Services (Lisbon, Portugal) (MobileHCI ’10). Association for Computing Machinery, New York, NY, USA, 153–156. https://doi.org/10.1145/1851600.1851626

Digital Library

[26]

Gierad Laput, Robert Xiao, Xiang “Anthony” Chen, Scott E. Hudson, and Chris Harrison. 2014. Skin Buttons: Cheap, Small, Low-Powered and Clickable Fixed-Icon Laser Projectors. In Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology (Honolulu, Hawaii, USA) (UIST ’14). Association for Computing Machinery, New York, NY, USA, 389–394. https://doi.org/10.1145/2642918.2647356

Digital Library

[27]

Nicolai Marquardt, Frederik Brudy, Can Liu, Ben Bengler, and Christian Holz. 2018. SurfaceConstellations: A Modular Hardware Platform for Ad-Hoc Reconfigurable Cross-Device Workspaces. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI ’18). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3173574.3173928

Digital Library

[28]

Nicolai Marquardt, Ricardo Jota, Saul Greenberg, and Joaquim A. Jorge. 2011. The Continuous Interaction Space: Interaction Techniques Unifying Touch and Gesture on and above a Digital Surface. In Human-Computer Interaction – INTERACT 2011, Pedro Campos, Nicholas Graham, Joaquim Jorge, Nuno Nunes, Philippe Palanque, and Marco Winckler(Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 461–476. https://doi.org/10.1007/978-3-642-23765-2_32

[29]

Tim Menzner, Travis Gesslein, Alexander Otte, and Jens Grubert. 2020. Above Surface Interaction for Multiscale Navigation in Mobile Virtual Reality. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). 372–381. https://doi.org/10.1109/VR46266.2020.00057

[30]

Alexandre Millette and Michael J. McGuffin. 2016. DualCAD: Integrating Augmented Reality with a Desktop GUI and Smartphone Interaction. In IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct). 21–26. https://doi.org/10.1109/ISMAR-Adjunct.2016.0030

[31]

Peter Mohr, Markus Tatzgern, Tobias Langlotz, Andreas Lang, Dieter Schmalstieg, and Denis Kalkofen. 2019. TrackCap: Enabling Smartphones for 3D Interaction on Mobile Head-Mounted Displays. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3290605.3300815

Digital Library

[32]

Meredith Ringel Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, Jacob O Wobbrock, 2014. Reducing legacy bias in gesture elicitation studies. interactions 21, 3 (2014), 40–45. https://doi.org/10.1145/2591689

Digital Library

[33]

Erwan Normand and Michael J. McGuffin. 2018. Enlarging a Smartphone with AR to Create a Handheld VESAD (Virtually Extended Screen-Aligned Display). In IEEE International Symposium on Mixed and Augmented Reality (ISMAR). 123–133. https://doi.org/10.1109/ISMAR.2018.00043

[34]

Anna Ostberg and Nada Matic. 2015. Hover Cursor: Improving Touchscreen Acquisition Of Small Targets With Hover-Enabled Pre-Selection. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI EA ’15). Association for Computing Machinery, New York, NY, USA, 1723–1728. https://doi.org/10.1145/2702613.2732903

Digital Library

[35]

Roman Rädle, Hans-Christian Jetter, Nicolai Marquardt, Harald Reiterer, and Yvonne Rogers. 2014. HuddleLamp: Spatially-Aware Mobile Displays for Ad-Hoc Around-the-Table Collaboration. In Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces (Dresden, Germany) (ITS ’14). Association for Computing Machinery, New York, NY, USA, 45–54. https://doi.org/10.1145/2669485.2669500

Digital Library

[36]

Hanae Rateau, Yosra Rekik, Edward Lank, and Laurent Grisoni. 2018. Ether-Toolbars: Evaluating Off-Screen Toolbars for Mobile Interaction. In 23rd International Conference on Intelligent User Interfaces (Tokyo, Japan) (IUI ’18). Association for Computing Machinery, New York, NY, USA, 487–495. https://doi.org/10.1145/3172944.3172978

Digital Library

[37]

Marcos Serrano, Barrett Ens, Xing-Dong Yang, and Pourang Irani. 2015. Gluey: Developing a Head-Worn Display Interface to Unify the Interaction Experience in Distributed Display Environments. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services (Copenhagen, Denmark) (MobileHCI ’15). Association for Computing Machinery, New York, NY, USA, 161–171. https://doi.org/10.1145/2785830.2785838

Digital Library

[38]

Marcos Serrano, Dale Hildebrandt, Sriram Subramanian, and Pourang Irani. 2014. Identifying Suitable Projection Parameters and Display Configurations for Mobile True-3D Displays. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services (Toronto, ON, Canada) (MobileHCI ’14). Association for Computing Machinery, New York, NY, USA, 135–143. https://doi.org/10.1145/2628363.2628375

Digital Library

[39]

Daniel Spelmezan, Caroline Appert, Olivier Chapuis, and Emmanuel Pietriga. 2013. Controlling Widgets with One Power-up Button. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST ’13). Association for Computing Machinery, New York, NY, USA, 71–74. https://doi.org/10.1145/2501988.2502025

Digital Library

[40]

Daniel Spelmezan, Caroline Appert, Olivier Chapuis, and Emmanuel Pietriga. 2013. Side Pressure for Bidirectional Navigation on Small Devices. In Proceedings of the 15th International Conference on Human-Computer Interaction with Mobile Devices and Services(Munich, Germany) (MobileHCI ’13). Association for Computing Machinery, New York, NY, USA, 11–20. https://doi.org/10.1145/2493190.2493199

Digital Library

[41]

Hemant Bhaskar Surale, Aakar Gupta, Mark Hancock, and Daniel Vogel. 2019. TabletInVR: Exploring the Design Space for Using a Multi-Touch Tablet in Virtual Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300243

Digital Library

[42]

Daniel Vogel and Patrick Baudisch. 2007. Shift: A Technique for Operating Pen-Based Interfaces Using Touch. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI ’07). Association for Computing Machinery, New York, NY, USA, 657–666. https://doi.org/10.1145/1240624.1240727

Digital Library

[43]

Dirk Wenig, Johannes Schöning, Alex Olwal, Mathias Oben, and Rainer Malaka. 2017. WatchThru: Expanding Smartwatch Displays with Mid-Air Visuals and Wrist-Worn Augmented Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 716–721. https://doi.org/10.1145/3025453.3025852

Digital Library

[44]

Fengyuan Zhu and Tovi Grossman. 2020. BISHARE: Exploring Bidirectional Interactions Between Smartphones and Head-Mounted Augmented Reality. In Proceedings of the CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–14. https://doi.org/10.1145/3313831.3376233

Digital Library

Cited By

Takeda KManabe H(2024)Screen Augmentation Technique Using AR Glasses and Smartphone without External SensorsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648664(1-5)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3648664
Chen CLu L(2024)Effects of input mode and screen layout on the user interface design of a smart home central control systemDisplays10.1016/j.displa.2024.102914(102914)Online publication date: Dec-2024
https://doi.org/10.1016/j.displa.2024.102914
Nagra ASohail MAmjad KAsif MAli S(2022)Stress Avoidance and Exercise recommendation Through CVS Posture Observation2022 International Conference on IT and Industrial Technologies (ICIT)10.1109/ICIT56493.2022.9989130(1-10)Online publication date: 3-Oct-2022
https://doi.org/10.1109/ICIT56493.2022.9989130
Show More Cited By

AR-enhanced Widgets for Smartphone-centric Interaction
1. Human-centered computing

Recommendations

Using a mobile device as an interface tool for HMD-based AR applications
ACE '06: Proceedings of the 2006 ACM SIGCHI international conference on Advances in computer entertainment technology

In this paper, we suggest that we can use our mobile devices as an effective interface for HMD-based augmented reality (AR) applications. These days most of us can possibly send short message with our mobile phones and watch the movies on our PDAs. ...
Mobile phone based AR scene assembly
MUM '05: Proceedings of the 4th international conference on Mobile and ubiquitous multimedia

In this paper we describe a mobile phone based Augmented Reality application for 3D scene assembly. Augmented Reality on mobile phones extends the interaction capabilities on such handheld devices. It adds a 6 DOF isomorphic interaction technique for ...
Dual Phone AR: Using a Second Phone as a Controller for Mobile Augmented Reality
UIST '20 Adjunct: Adjunct Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology

Mobile Augmented Reality applications have become increasingly popular, however the possible interactions with AR content are largely limited to on-screen gestures and spatial movement. There has been a renewed interest in designing interaction methods ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MobileHCI '21: Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction

September 2021

637 pages

ISBN:9781450383288

DOI:10.1145/3447526

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

MobileHCI '21

Sponsor:

SIGCHI

MobileHCI '21: 23rd International Conference on Mobile Human-Computer Interaction

September 27 - October 1, 2021

Toulouse & Virtual, France

Acceptance Rates

Overall Acceptance Rate 202 of 906 submissions, 22%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
372
Total Downloads

Downloads (Last 12 months)78
Downloads (Last 6 weeks)7

Reflects downloads up to 16 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Takeda KManabe H(2024)Screen Augmentation Technique Using AR Glasses and Smartphone without External SensorsExtended Abstracts of the CHI Conference on Human Factors in Computing Systems10.1145/3613905.3648664(1-5)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613905.3648664
Chen CLu L(2024)Effects of input mode and screen layout on the user interface design of a smart home central control systemDisplays10.1016/j.displa.2024.102914(102914)Online publication date: Dec-2024
https://doi.org/10.1016/j.displa.2024.102914
Nagra ASohail MAmjad KAsif MAli S(2022)Stress Avoidance and Exercise recommendation Through CVS Posture Observation2022 International Conference on IT and Industrial Technologies (ICIT)10.1109/ICIT56493.2022.9989130(1-10)Online publication date: 3-Oct-2022
https://doi.org/10.1109/ICIT56493.2022.9989130
Zhang XHe WBillinghurst MLiu DYang LFeng SLiu Y(2022)Usability of Cross-Device Interaction Interfaces for Augmented Reality in Physical TasksInternational Journal of Human–Computer Interaction10.1080/10447318.2022.216053740:9(2361-2379)Online publication date: 28-Dec-2022
https://doi.org/10.1080/10447318.2022.2160537

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten