research-article

Public Access

Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environment

Authors:

Adam Sinclair Williams,

Francisco Raul OrtegaAuthors Info & Claims

VRST '22: Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology

Article No.: 17, Pages 1 - 10

https://doi.org/10.1145/3562939.3565637

Published: 29 November 2022 Publication History

All formats PDF

Abstract

As augmented reality (AR) technology and hardware become more mature and affordable, researchers have been exploring more intuitive and discoverable interaction techniques for immersive environments. This paper investigates multimodal interaction for 3D object manipulation in a multi-object AR environment. To identify the user-defined gestures, we conducted an elicitation study involving 24 participants and 22 referents using an augmented reality headset. It yielded 528 proposals and generated a winning gesture set with 25 gestures after binning and ranking all gesture proposals. We found that for the same task, the same gesture was preferred for both one and two-object manipulation, although both hands were used in the two-object scenario. We present the gestures and speech results, and the differences compared to similar studies in a single object AR environment. The study also explored the association between speech expressions and gesture stroke during object manipulation, which could improve the recognizer efficiency in augmented reality headsets.

References

[1]

Muhammad Zeeshan Baig and Manolya Kavakli. 2018. Qualitative analysis of a multimodal interface system using speech/gesture. In 2018 13th IEEE Conference on Industrial Electronics and Applications (ICIEA). IEEE, IEEE, Wuhan, China, 2811–2816.

[2]

R Balakrishnan and K Hinckley. 2000. Symmetric bimanual interaction. GROUP ACM SIGCHI Int. Conf. Support. Group Work (2000).

Digital Library

[3]

Marie-Luce Bourguet and Akio Ando. 1998. Synchronization of speech and hand gestures during multimodal human-computer interaction. In CHI 98 Conference Summary on Human Factors in Computing Systems. ACM, 241–242.

Digital Library

[4]

Aurélie Cohé and Martin Hachet. 2012. Understanding User Gestures for Manipulating 3D Objects from Touchscreen Inputs. In Proceedings of Graphics Interface 2012 (Toronto, Ontario, Canada) (GI ’12). Canadian Information Processing Society, Toronto, Ont., Canada, Canada, 157–164. http://dl.acm.org/citation.cfm?id=2305276.2305303

Digital Library

[5]

Andreea Danielescu and David Piorkowski. 2022. Iterative design of gestures during elicitation: Understanding the role of increased production. https://arxiv.org/abs/2104.04685

[6]

Niloofar Dezfuli, Mohammadreza Khalilbeigi, Max Mühlhäuser, and David Geerts. 2011. A Study on Interpersonal Relationships for Social Interactive Television. In Proceedings of the 9th European Conference on Interactive TV and Video (Lisbon, Portugal) (EuroITV ’11). Association for Computing Machinery, New York, NY, USA, 21–24. https://doi.org/10.1145/2000119.2000123

Digital Library

[7]

Robert J.K. Jacob, Audrey Girouard, Leanne M. Hirshfield, Michael S. Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-Based Interaction: A Framework for Post-WIMP Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Florence, Italy) (CHI ’08). Association for Computing Machinery, New York, NY, USA, 201–210. https://doi.org/10.1145/1357054.1357089

Digital Library

[8]

A A Karpov and R M Yusupov. 2018. Multimodal Interfaces of Human–Computer Interaction. Her. Russ. Acad. Sci. 88, 1 (Jan. 2018), 67–74.

[9]

Sumbul Khan and Bige Tunçer. 2019. Gesture and speech elicitation for 3D CAD modeling in conceptual design. Automation in Construction 106 (2019), 102847.

[10]

Minkyung Lee and Mark Billinghurst. 2008. A Wizard of Oz Study for an AR Multimodal Interface. In Proceedings of the 10th International Conference on Multimodal Interfaces (Chania, Crete, Greece) (ICMI ’08). Association for Computing Machinery, New York, NY, USA, 249–256. https://doi.org/10.1145/1452392.1452444

Digital Library

[11]

David Mcneill. 2005. Gesture and Thought. the University of Chicago Press, USA. https://doi.org/10.7208/chicago/9780226514642.001.0001

[12]

Meredith Ringel Morris. 2012. Web on the Wall: Insights from a Multimodal Interaction Elicitation Study. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces(Cambridge, Massachusetts, USA) (ITS ’12). ACM, New York, NY, USA, 95–104. https://doi.org/10.1145/2396636.2396651

Digital Library

[13]

Meredith Ringel Morris, Andreea Danielescu, Steven Drucker, Danyel Fisher, Bongshin Lee, M c Schraefel, and Jacob O Wobbrock. 2014. Reducing Legacy Bias in Gesture Elicitation Studies. Interactions 21, 3 (May 2014), 40–45.

Digital Library

[14]

Meredith Ringel Morris, Jacob O Wobbrock, and Andrew D Wilson. 2010. Understanding users’ preferences for surface gestures. In Proceedings of graphics interface 2010. Canadian Information Processing Society, 261–268.

[15]

Miguel A. Nacenta, Yemliha Kamber, Yizhou Qiang, and Per Ola Kristensson. 2013. Memorability of Pre-Designed and User-Defined Gesture Sets. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Paris, France) (CHI ’13). Association for Computing Machinery, New York, NY, USA, 1099–1108. https://doi.org/10.1145/2470654.2466142

Digital Library

[16]

Michael Nielsen, Moritz Störring, Thomas B. Moeslund, and Erik Granum. 2004. A Procedure for Developing Intuitive and Ergonomic Gesture Interfaces for HCI. In Gesture-Based Communication in Human-Computer Interaction, Antonio Camurri and Gualtiero Volpe (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg, 409–420.

[17]

Francisco R. Ortega, Alain Galvan, Katherine Tarre, Armando Barreto, Naphtali Rishe, Jonathan Bernal, Ruben Balcazar, and Jason-Lee Thomas. 2017. Gesture elicitation for 3D travel via multi-touch and mid-Air systems for procedurally generated pseudo-universe. In 2017 IEEE Symposium on 3D User Interfaces (3DUI). 144–153. https://doi.org/10.1109/3DUI.2017.7893331

[18]

Tran Pham, Jo Vermeulen, Anthony Tang, and Lindsay MacDonald Vermeulen. 2018. Scale Impacts Elicited Gestures for Manipulating Holograms: Implications for AR Gesture Design. In Proceedings of the 2018 Designing Interactive Systems Conference. ACM, 227–240.

Digital Library

[19]

Thammathip Piumsomboon, Adrian Clark, Mark Billinghurst, and Andy Cockburn. 2013. User-Defined Gestures for Augmented Reality. In CHI ’13 Extended Abstracts on Human Factors in Computing Systems (Paris, France) (CHI EA ’13). Association for Computing Machinery, New York, NY, USA, 955–960. https://doi.org/10.1145/2468356.2468527

Digital Library

[20]

Jaime Ruiz, Yang Li, and Edward Lank. 2011. User-Defined Motion Gestures for Mobile Interaction(CHI ’11). Association for Computing Machinery, New York, NY, USA, 197–206. https://doi.org/10.1145/1978942.1978971

Digital Library

[21]

Giulia Wally Scurati, Michele Gattullo, Michele Fiorentino, Francesco Ferrise, Monica Bordegoni, and Antonio Emmanuele Uva. 2018. Converting maintenance actions into standard symbols for Augmented Reality applications in Industry 4.0. Computers in Industry 98(2018), 68–79. https://doi.org/10.1016/j.compind.2018.02.001

Digital Library

[22]

Teddy Seyed, Chris Burns, Mario Costa Sousa, Frank Maurer, and Anthony Tang. 2012. Eliciting Usable Gestures for Multi-Display Environments. In Proceedings of the 2012 ACM International Conference on Interactive Tabletops and Surfaces (Cambridge, Massachusetts, USA) (ITS ’12). Association for Computing Machinery, New York, NY, USA, 41–50. https://doi.org/10.1145/2396636.2396643

Digital Library

[23]

SHENG KAI TANG and David Coulter. 2022. Start gesture - mixed reality. https://docs.microsoft.com/en-us/windows/mixed-reality/design/system-gesture

[24]

Theophanis Tsandilas. 2018. Fallacies of Agreement: A Critical Review of Consensus Assessment Methods for Gesture Elicitation. ACM Trans. Comput. Hum. Interact. 25, 3 (June 2018), 18.

Digital Library

[25]

Radu-Daniel Vatavu and Jacob O. Wobbrock. 2015. Formalizing Agreement Analysis for Elicitation Studies: New Measures, Significance Test, and Toolkit. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI ’15). Association for Computing Machinery, New York, NY, USA, 1325–1334. https://doi.org/10.1145/2702123.2702223

Digital Library

[26]

Radu-Daniel Vatavu and Jacob O. Wobbrock. 2022. Clarifying Agreement Calculations and Analysis for End-User Elicitation Studies. ACM Trans. Comput.-Hum. Interact. 29, 1, Article 5 (jan 2022), 70 pages. https://doi.org/10.1145/3476101

Digital Library

[27]

Santiago Villarreal-Narvaez, Jean Vanderdonckt, Radu-Daniel Vatavu, and Jacob A Wobbrock. 2020. A Systematic Review of Gesture Elicitation Studies: What Can We Learn from 216 Studies. In Proceedings of ACM Int. Conf. on Designing Interactive Systems (DIS’20). ACM Press, Eindhoven, NA.

Digital Library

[28]

Panagiotis Vogiatzidakis and Panayiotis Koutsabasis. 2022. ‘Address and command’: Two-handed mid-air interactions with multiple home devices. International Journal of Human-Computer Studies 159 (2022), 102755. https://doi.org/10.1016/j.ijhcs.2021.102755

Digital Library

[29]

Adam S. Williams, Jason Garcia, and Francisco Ortega. 2020. Understanding Multimodal User Gesture and Speech Behavior for Object Manipulation in Augmented Reality Using Elicitation. IEEE Transactions on Visualization and Computer Graphics 26, 12(2020), 3479–3489. https://doi.org/10.1109/TVCG.2020.3023566

[30]

Adam S Williams and Francisco Ortega. 2020. Insights on visual aid and study design for gesture interaction in limited sensor range Augmented Reality devices. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). 19–22. https://doi.org/10.1109/VRW50115.2020.00286

[31]

Adam S. Williams and Francisco R. Ortega. 2020. Evolutionary Gestures: When a Gesture is Not Quite Legacy Biased. Interactions 27, 5 (sep 2020), 50–53. https://doi.org/10.1145/3412499

Digital Library

[32]

Adam S. Williams and Francisco R. Ortega. 2020. Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation. Proc. ACM Hum.-Comput. Interact. 4, ISS, Article 202 (nov 2020), 21 pages. https://doi.org/10.1145/3427330

Digital Library

[33]

Adam S. Williams and Francisco R. Ortega. 2021. A concise guide to elicitation methodology. https://arxiv.org/abs/2105.12865

[34]

Markus L Wittorf and Mikkel R Jakobsen. 2016. Eliciting Mid-Air Gestures for Wall-Display Interaction. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction (Gothenburg, Sweden) (NordiCHI ’16). ACM, New York, NY, USA, 3:1–3:4.

Digital Library

[35]

Jacob O. Wobbrock, Htet Htet Aung, Brandon Rothrock, and Brad A. Myers. 2005. Maximizing the Guessability of Symbolic Input. In CHI ’05 Extended Abstracts on Human Factors in Computing Systems (Portland, OR, USA) (CHI EA ’05). Association for Computing Machinery, New York, NY, USA, 1869–1872. https://doi.org/10.1145/1056808.1057043

Digital Library

[36]

Jacob O Wobbrock, Meredith Ringel Morris, and Andrew D Wilson. 2009. User-defined Gestures for Surface Computing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Boston, MA, USA) (CHI ’09). ACM, New York, NY, USA, 1083–1092.

Digital Library

[37]

Xiaoyan Zhou, Adam S. Williams, and Francisco R. Ortega. 2022. Towards Establishing Consistent Proposal Binning Methods for Unimodal and Multimodal Interaction Elicitation Studies. In Human-Computer Interaction. Theoretical Approaches and Design Methods, Masaaki Kurosu (Ed.). Springer International Publishing, Cham, 356–368.

Cited By

Solomashenko AAfanaseva OShishova MGulianskii ISobolnikov SPetrov N(2025)Industrial applications of AR headsets: a review of the devices and experienceLight: Advanced Manufacturing10.37188/lam.2025.0236(1)Online publication date: 2025
https://doi.org/10.37188/lam.2025.023
Borhani ZBorhani FOrtega F(2024)Experiencing Gravitational Red-Shiftting in Virtual Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00028(133-137)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00028
Cheng YWu ZXiao R(2024)Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.338726912(64958-64979)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387269
Show More Cited By

Index Terms

Eliciting Multimodal Gesture+Speech Interactions in a Multi-Object Augmented Reality Environment
1. Computer systems organization
  1. Dependable and fault-tolerant systems and networks
    1. Redundancy
  2. Embedded and cyber-physical systems
    1. Embedded systems
    2. Robotics
2. Networks
  1. Network properties
    1. Network reliability

Recommendations

Understanding Gesture and Speech Multimodal Interactions for Manipulation Tasks in Augmented Reality Using Unconstrained Elicitation
ISS

This research establishes a better understanding of the syntax choices in speech interactions and of how speech, gesture, and multimodal gesture and speech interactions are produced by users in unconstrained object manipulation environments using ...
Multimodal augmented reality: the norm rather than the exception
MVAR '16: Proceedings of the 2016 workshop on Multimodal Virtual and Augmented Reality

Augmented reality (AR) is commonly seen as a technology that overlays virtual imagery onto a participant's view of the world. In line with this, most AR research is focused on what we see. In this paper, we challenge this focus on vision and make a case ...
Haptics in Augmented Reality
ICMCS '99: Proceedings of the IEEE International Conference on Multimedia Computing and Systems - Volume 2

An augmented reality system merges synthetic sensory information into a user's perception of a three-dimensional environment. An important performance goal for an augmented reality system is that the user perceives a single seamless environment. In most ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

VRST '22: Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology

November 2022

466 pages

ISBN:9781450398893

DOI:10.1145/3562939

Editors:
Takafumi Koike
Hosei University, Japan
,
Naoya Koizumi
The University of Electro-Communications, Japan
,
Gerd Bruder
University of Central Florida, USA
,
Daniel Roth
Friedrich-Alexander-Universität Erlangen-Nürnberg, Germany
,
Kazuki Takashima
Tohoku University, Japan
,
Takefumi Hiraki
University of Tsukuba, Japan
,
Yuki Ban
The University of Tokyo, Japan
,
Michal Piovarci
Institute of Science and Technology Austria, Austria

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 November 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

NSF (National Science Foundation)

Conference

VRST '22

Sponsor:

VRST '22: 28th ACM Symposium on Virtual Reality Software and Technology

November 29 - December 1, 2022

Tsukuba, Japan

Acceptance Rates

Overall Acceptance Rate 66 of 254 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
620
Total Downloads

Downloads (Last 12 months)327
Downloads (Last 6 weeks)34

Reflects downloads up to 05 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Solomashenko AAfanaseva OShishova MGulianskii ISobolnikov SPetrov N(2025)Industrial applications of AR headsets: a review of the devices and experienceLight: Advanced Manufacturing10.37188/lam.2025.0236(1)Online publication date: 2025
https://doi.org/10.37188/lam.2025.023
Borhani ZBorhani FOrtega F(2024)Experiencing Gravitational Red-Shiftting in Virtual Reality2024 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW62533.2024.00028(133-137)Online publication date: 16-Mar-2024
https://doi.org/10.1109/VRW62533.2024.00028
Cheng YWu ZXiao R(2024)Exploring Methods to Optimize Gesture Elicitation Studies: A Systematic Literature ReviewIEEE Access10.1109/ACCESS.2024.338726912(64958-64979)Online publication date: 2024
https://doi.org/10.1109/ACCESS.2024.3387269
Qin YHe WZhang XBillinghurst MZhang YDong J(2024)Usability of Pseudo-Haptic Feedback for Manual Precise Manipulation of Objects in Augmented RealityInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2400817(1-16)Online publication date: 18-Sep-2024
https://doi.org/10.1080/10447318.2024.2400817
Mathew RTigwell GPeiris R(2024)Deaf and Hard of Hearing People’s Perspectives on Augmented Reality Interfaces for Improving the Accessibility of Smart SpeakersUniversal Access in Human-Computer Interaction10.1007/978-3-031-60881-0_21(334-357)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1007/978-3-031-60881-0_21

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten