Cited By
View all- Kianpisheh MMariakakis ATruong K(2024)exHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435008:1(1-30)Online publication date: 6-Mar-2024
Investigating Cross-Modal Approaches for Evaluating Error Acceptability of a Recognition-Based Input Technique
Authors: 
Jay Henderson, 
Tanya R. Jonker, Edward Lank, 
Daniel Wigdor, Ben LafreniereAuthors Info & Claims
Jay Henderson, Tanya R. Jonker, Edward Lank, Daniel Wigdor, Ben LafreniereAuthors Info & ClaimsArticle No.: 15, Pages 1 - 24
Abstract
Emerging input techniques that rely on sensing and recognition can misinterpret a user's intention, resulting in errors and, potentially, a negative user experience. To enhance the development of such input techniques, it is valuable to understand implications of these errors, but they can very costly to simulate. Through two controlled experiments, this work explores various low-cost methods for evaluating error acceptability of freehand mid-air gestural input in virtual reality. Using a gesture-driven game and a drawing application, the first experiment elicited error characteristics through text descriptions, video demonstrations, and a touchscreen-based interactive simulation. The results revealed that video effectively conveyed the dynamics of errors, whereas the interactive modalities effectively reproduced the user experience of effort and frustration. The second experiment contrasts the interactive touchscreen simulation with the target modality - a full VR simulation - and highlights the relative costs and benefits for assessment in an alternative, but still interactive, modality. These findings introduce a spectrum of low-cost methods for evaluating recognition-based errors in VR and a series of characteristics that can be understood in each.
Supplementary Material
Supplemental movie, appendix, image and software files for, Investigating Cross-Modal Approaches for Evaluating Error Acceptability of a Recognition-Based Input Technique
- Download
- 96.39 KB
References
[1]
Fraser Anderson, Tovi Grossman, Justin Matejka, and George Fitzmaurice. 2013. YouMove: Enhancing Movement Training with an Augmented Reality Mirror. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 311--320. https://doi.org/10.1145/2501988.2502045
[2]
Ravin Balakrishnan and I. Scott MacKenzie. 1997. Performance Differences in the Fingers, Wrist, and Forearm in Computer Input Control. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA) (CHI '97). Association for Computing Machinery, New York, NY, USA, 303--310. https://doi.org/10.1145/258549.258764
[3]
Olivier Bau and Wendy E. Mackay. 2008. OctoPocus: a dynamic guide for learning gesture-based command sets. In Proceedings of the 21st annual ACM symposium on User interface software and technology (Monterey, CA, USA, 2008-10-19) (UIST '08). Association for Computing Machinery, 37--46. https://doi.org/10.1145/1449715.1449724
[4]
Michel Beaudouin-Lafon and Wendy E. Mackay. 2012. Prototyping Tools and Techniques. In The Human Computer Interaction Handbook (3 ed.), Julie A. Jacko (Ed.). CRC Press, Boca Raton, FL, Chapter 47, 1081--1104. https://doi.org/10.1201/b11963-55
[5]
Marion Buchenau and Jane Fulton Suri. [n.d.]. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques (New York City, New York, USA, 2000-08-01) (DIS '00). Association for Computing Machinery, 424--433. https://doi.org/10.1145/347642.347802
[6]
Nils Dahlbäck, Arne Jönsson, and Lars Ahrenberg. 1993. Wizard of Oz Studies: Why and How. In Proceedings of the 1st International Conference on Intelligent User Interfaces (Orlando, Florida, USA) (IUI '93). Association for Computing Machinery, New York, NY, USA, 193--200. https://doi.org/10.1145/169891.169968
[7]
Fred Davis. 1989. Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly 13 (09 1989), 319-. https://doi.org/10.2307/249008
[8]
William Delamare, Thomas Janssoone, Céline Coutrix, and Laurence Nigay. 2016. Designing 3D Gesture Guidance: Visual Feedback and Feedforward Design Options. In Proceedings of the International Working Conference on Advanced Visual Interfaces (Bari, Italy) (AVI '16). Association for Computing Machinery, New York, NY, USA, 152--159. https://doi.org/10.1145/2909132.2909260
[9]
P. Dourish. 2004. Where the Action Is: The Foundations of Embodied Interaction. MIT Press. https://books.google.ca/books?id=-TRWc0PA9e4C
[10]
Graham Dove, Kim Halskov, Jodi Forlizzi, and John Zimmerman. [n.d.]. UX Design Innovation: Challenges for Working with Machine Learning As a Design Material. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (New York, NY, USA, 2017) (CHI '17). ACM, 278--288. https://doi.org/10.1145/3025453.3025739
[11]
Pelle Ehn and Morten Kyng. 1992. Cardboard Computers: Mocking-It-up or Hands-on the Future. In Design at Work: Cooperative Design of Computer Systems. L. Erlbaum Associates Inc., USA, 169--196.
[12]
World Leaders in Research-Based User Experience. 1994. Guerrilla HCI: Article by Jakob Nielsen. https://www.nngroup.com/articles/guerrilla-hci/ Library Catalog: www.nngroup.com.
[13]
World Leaders in Research-Based User Experience. 2009. Discount Usability: 20 Years. https://www.nngroup.com/articles/discount-usability-20-years/ Library Catalog: www.nngroup.com.
[14]
Dustin Freeman, Hrvoje Benko, Meredith Ringel Morris, and Daniel Wigdor. [n.d.]. ShadowGuides: visualizations for in-situ learning of multi-touch and whole-hand gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (Banff, Alberta, Canada, 2009-11-23) (ITS '09). Association for Computing Machinery, 165--172. https://doi.org/10.1145/1731903.1731935
[15]
Dustin Freeman, Hrvoje Benko, Meredith Ringel Morris, and Daniel Wigdor. 2009. ShadowGuides: Visualizations for in-Situ Learning of Multi-Touch and Whole-Hand Gestures. In Proceedings of the ACM International Conference on Interactive Tabletops and Surfaces (Banff, Alberta, Canada) (ITS '09). Association for Computing Machinery, New York, NY, USA, 165--172. https://doi.org/10.1145/1731903.1731935
[16]
Saul Greenberg and Chester Fitchett. [n.d.]. Phidgets: easy development of physical interfaces through physical widgets. In Proceedings of the 14th annual ACM symposium on User interface software and technology (Orlando, Florida, 2001-11-11) (UIST '01). Association for Computing Machinery, 209--218. https://doi.org/10.1145/502348.502388
[17]
Carl Gutwin, Andy Cockburn, and Benjamin Lafreniere. 2015. Testing the Rehearsal Hypothesis with Two FastTap Interfaces. 223--231.
[18]
Sandra G. Hart. 2006. Nasa-Task Load Index (NASA-TLX); 20 Years Later. Proceedings of the Human Factors and Ergonomics Society Annual Meeting 50, 9 (2006), 904--908. https://doi.org/10.1177/154193120605000909 arXiv:https://doi.org/10.1177/154193120605000909
[19]
Björn Hartmann, Leith Abdulla, Manas Mittal, and Scott R. Klemmer. 2007. Authoring Sensor-Based Interactions by Demonstration with Direct Manipulation and Pattern Recognition. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (San Jose, California, USA) (CHI '07). Association for Computing Machinery, New York, NY, USA, 145--154. https://doi.org/10.1145/1240624.1240646
[20]
Jay Henderson, Sachi Mizobuchi, Wei Li, and Edward Lank. 2019. Exploring Cross-Modal Training via Touch to Learn a Mid-Air Marking Menu Gesture Set. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 8, 9 pages. https://doi.org/10.1145/3338286.3340119
[21]
Ankit Kamal, Yang Li, and Edward Lank. [n.d.]. Teaching motion gestures via recognizer feedback. In Proceedings of the 19th international conference on Intelligent User Interfaces (Haifa, Israel, 2014-02-24) (IUI '14). Association for Computing Machinery, 73--82. https://doi.org/10.1145/2557500.2557521
[22]
Ankit Kamal, Yang Li, and Edward Lank. 2014. Teaching Motion Gestures via Recognizer Feedback. In Proceedings of the 19th International Conference on Intelligent User Interfaces (Haifa, Israel) (IUI '14). Association for Computing Machinery, New York, NY, USA, 73--82. https://doi.org/10.1145/2557500.2557521
[23]
Keiko Katsuragawa, Ankit Kamal, and Edward Lank. 2017. Effect of motion-gesture recognizer error pattern on user workload and behavior. In International Conference on Intelligent User Interfaces, Proceedings IUI. https://doi.org/10.1145/3025171.3025234
[24]
Keiko Katsuragawa, Ankit Kamal, Qi Feng Liu, Matei Negulescu, and Edward Lank. 2019. Bi-Level Thresholding: Analyzing the Effect of Repeated Errors in Gesture Input. ACM Trans. Interact. Intell. Syst. 9, 2-3, Article 15 (April 2019), 30 pages. https://doi.org/10.1145/3181672
[25]
Matthew Kay, Shwetak N. Patel, and Julie A. Kientz. 2015. How good is 85%? A survey tool to connect classifier evaluation to acceptability of accuracy. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/2702123.2702603
[26]
Matthew Kay and Jacob O. Wobbrock. 2020. Aligned Rank Transform for Nonparametric Factorial ANOVAs. https://doi.org/10.5281/zenodo.594511
[27]
Rafal Kocielnik, Saleema Amershi, and Paul N. Bennett. 2019. Will you accept an imperfect AI? Exploring Designs for Adjusting End-user Expectations of AI Systems. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3290605.3300641
[28]
Matthias Kranz, Paul Holleis, and Albrecht Schmidt. 2010. Embedded Interaction: Interacting with the Internet of Things. IEEE Internet Computing 14, 2 (2010), 46--53. https://doi.org/10.1109/MIC.2009.141
[29]
Ben Lafreniere, Tanya R. Jonker, Stephanie Santosa, Mark Parent, Michael Glueck, Tovi Grossman, Hrvoje Benko, and Daniel Wigdor. 2021. False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error Preference. In The 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '21). Association for Computing Machinery, New York, NY, USA, 54--68. https://doi.org/10.1145/3472749.3474735
[30]
David Ledo, Fraser Anderson, Ryan Schmidt, Lora Oehlberg, Saul Greenberg, and Tovi Grossman. [n.d.]. Pineal: Bringing Passive Objects to Life with Embedded Mobile Devices. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA, 2017-05-02) (CHI '17). Association for Computing Machinery, 2583--2593. https://doi.org/10.1145/3025453.3025652
[31]
David Ledo, Jo Vermeulen, Sheelagh Carpendale, Saul Greenberg, Lora Oehlberg, and Sebastian Boring. [n.d.]. Astral: Prototyping Mobile and Smart Object Interactive Behaviours Using Familiar Applications. In Proceedings of the 2019 on Designing Interactive Systems Conference (San Diego, CA, USA, 2019-06-18) (DIS '19). Association for Computing Machinery, 711--724. https://doi.org/10.1145/3322276.3322329
[32]
Yang Li. [n.d.]. Protractor: a fast and accurate gesture recognizer. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Atlanta, Georgia, USA, 2010-04-10) (CHI '10). Association for Computing Machinery, 2169--2172. https://doi.org/10.1145/1753326.1753654
[33]
Microsoft. 2020. Gestures - Mixed Reality Toolkit Documentation. https://microsoft.github.io/MixedRealityToolkit-Unity/Documentation/Input/Gestures.html
[34]
William Odom, John Zimmerman, Scott Davidoff, Jodi Forlizzi, Anind K. Dey, and Min Kyung Lee. [n.d.]. A fieldwork of the future with user enactments. In Proceedings of the Designing Interactive Systems Conference (Newcastle Upon Tyne, United Kingdom, 2012-06-11) (DIS 12). Association for Computing Machinery, 338--347. https://doi.org/10.1145/2317956.2318008
[35]
Quentin Roy, Futian Zhang, and Daniel Vogel. 2019. Automation accuracy is good, but high controllability may be better. In Conference on Human Factors in Computing Systems - Proceedings. https://doi.org/10.1145/3290605.3300750
[36]
Dean Rubine. 1991. Specifying Gestures by Example. In Proceedings of the 18th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH '91). Association for Computing Machinery, New York, NY, USA, 329--337. https://doi.org/10.1145/122718.122753
[37]
Valkyrie Savage, Colin Chang, and Björn Hartmann. 2013. Sauron: Embedded Single-Camera Sensing of Printed Physical User Interfaces. In Proceedings of the 26th Annual ACM Symposium on User Interface Software and Technology (St. Andrews, Scotland, United Kingdom) (UIST '13). Association for Computing Machinery, New York, NY, USA, 447--456. https://doi.org/10.1145/2501988.2501992
[38]
Rajinder Sodhi, Hrvoje Benko, and Andrew Wilson. 2012. LightGuide: Projected Visualizations for Hand Movement Guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Austin, Texas, USA) (CHI '12). Association for Computing Machinery, New York, NY, USA, 179--188. https://doi.org/10.1145/2207676.2207702
[39]
Seiichi Uchida and Hiroaki Sakoe. 2005. A survey of elastic matching techniques for handwritten character recognition. IEICE transactions on information and systems 88, 8 (2005), 1781--1790.
[40]
Viswanath Venkatesh and Fred Davis. 2000. A Theoretical Extension of the Technology Acceptance Model: Four Longitudinal Field Studies. Management Science 46 (02 2000), 186--204. https://doi.org/10.1287/mnsc.46.2.186.11926
[41]
Daniel S. Weld and Gagan Bansal. 2019. The Challenge of Crafting Intelligible Intelligence. Commun. ACM 62, 6 (May 2019), 70--79. https://doi.org/10.1145/3282486
[42]
Daniel Wigdor and Dennis Wixon. 2011. Brave NUI World: Designing Natural User Interfaces for Touch and Gesture (1st ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA.
[43]
Andrew D. Wilson. 2006. Robust Computer Vision-Based Detection of Pinching for One and Two-Handed Gesture Input. In Proceedings of the 19th Annual ACM Symposium on User Interface Software and Technology (Montreux, Switzerland) (UIST 06). Association for Computing Machinery, New York, NY, USA, 255--258. https://doi.org/10.1145/1166253.1166292
[44]
Jacob O. Wobbrock, Leah Findlater, Darren Gergle, and James J. Higgins. 2011. The Aligned Rank Transform for Nonparametric Factorial Analyses Using Only Anova Procedures. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). Association for Computing Machinery, New York, NY, USA, 143--146. https://doi.org/10.1145/1978942.1978963
[45]
J. Zimmerman. 2005. Video sketches: exploring pervasive computing interaction designs. IEEE Pervasive Computing 4, 4 (2005), 91--94.
Index Terms
- Investigating Cross-Modal Approaches for Evaluating Error Acceptability of a Recognition-Based Input Technique
Recommendations
Exploring the use of hand-to-face input for interacting with head-worn displays
CHI '14: Proceedings of the SIGCHI Conference on Human Factors in Computing SystemsWe propose the use of Hand-to-Face input, a method to interact with head-worn displays (HWDs) that involves contact with the face. We explore Hand-to-Face interaction to find suitable techniques for common mobile tasks. We evaluate this form of ...
False Positives vs. False Negatives: The Effects of Recovery Time and Cognitive Costs on Input Error Preference
UIST '21: The 34th Annual ACM Symposium on User Interface Software and TechnologyExisting approaches to trading off false positive versus false negative errors in input recognition are based on imprecise ideas of how these errors affect user experience that are unlikely to hold for all situations. To inform dynamic approaches to ...
Investigating current techniques for opposite-hand smartwatch interaction
MobileHCI '17: Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and ServicesThe small display size of smartwatches creates a challenge for touch input, which is still the interaction technique of choice. Researchers and producers have started to investigate alternative interaction techniques. Apple and Samsung, for example, ...
Comments
Information & Contributors
Information
Published In
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 March 2022
Published in IMWUT Volume 6, Issue 1
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 248Total Downloads
- Downloads (Last 12 months)79
- Downloads (Last 6 weeks)6
Reflects downloads up to 10 Aug 2024
Other Metrics
Citations
Cited By
View all- Kianpisheh MMariakakis ATruong K(2024)exHARProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36435008:1(1-30)Online publication date: 6-Mar-2024
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in