Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Evaluating the Performance of Hand-Based Probabilistic Text Input Methods on a Mid-Air Virtual Qwerty Keyboard

Published: 04 October 2023 Publication History

Abstract

Integrated hand-tracking on modern virtual reality (VR) headsets can be readily exploited to deliver mid-air virtual input surfaces for text entry. These virtual input surfaces can closely replicate the experience of typing on a Qwerty keyboard on a physical touchscreen, thereby allowing users to leverage their pre-existing typing skills. However, the lack of passive haptic feedback, unconstrained user motion, and potential tracking inaccuracies or observability issues encountered in this interaction setting typically degrades the accuracy of user articulations. We present a comprehensive exploration of error-tolerant probabilistic hand-based input methods to support effective text input on a mid-air virtual Qwerty keyboard. Over three user studies we examine the performance potential of hand-based text input under both gesture and touch typing paradigms. We demonstrate typical entry rates in the range of 20 to 30 wpm and average peak entry rates of 40 to 45 wpm.

References

[1]
J. Adhikary and K. Vertanen. Text Entry in Virtual Environments using Speech and a Midair Keyboard. IEEE Transactions on Visualization and Computer Graphics, 27 (5): pp. 2648–2658, May 2021. Conference Name: IEEE Transactions on Visualization and Computer Graphics. 2, 3.
[2]
J. Adhikary and K. Vertanen. “Typing on Midair Virtual Keyboards: Exploring Visual Designs and Interaction Styles”. In C. Ardito, R. Lanzilotti, A. Malizia, H. Petrie, A. Piccinno, G. Desolda, and K. Inkpen, eds., Human-Computer Interaction - INTERACT 2021, pp. 132–151. Cham: Springer International Publishing, 2021. 3.
[3]
O. Alsharif, T. Ouyang, F. Beaufays, S. Zhai, T. Breuel, and J. Schalkwyk. Long short term memory neural network for keyboard gesture decoding. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2076–2080, 2015. 4.
[4]
R. Arora, R. H. Kazi, F. Anderson, T. Grossman, K. Singh, and G. Fitzmaurice. Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pp. 5643–5654. ACM, New York, NY, USA, 2017. 1.
[5]
G. Benoit, G. M. Poor, and A. Jude. Bimanual Word Gesture Keyboards for Mid-Air Gestures. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA'17, pp. 1500–1507. Association for Computing Machinery, New York, NY, USA, 2017. 3.
[6]
S. Chen, J. Wang, S. Guerra, N. Mittal, and S. Prakkamakul. Exploring Word-gesture Text Entry Techniques in Virtual Reality. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems, CHI EA '19, pp. 1–6. Association for Computing Machinery, New York, NY, USA, May 2019. 2, 3.
[7]
T. J. Dube and A. S. Arif. “Text Entry in Virtual Reality: A Comprehensive Review of the Literature”. In M. Kurosu, ed., Human-Computer Interaction. Recognition and Interaction Technologies, Lecture Notes in Computer Science, pp. 419–437. Cham: Springer International Publishing, 2019. 2.
[8]
T. J. Dube and A. S. Arif. Ultrasonic Keyboard: A Mid-Air Virtual Qwerty with Ultrasonic Feedback for Virtual Reality. In Proceedings of the Seventeenth International Conference on Tangible, Embedded, and Embodied Interaction, TEI '23. Association for Computing Machinery, New York, NY, USA, 2023. 3.
[9]
T. J. Dube, K. Johnson, and A. S. Arif. Shapeshifter: Gesture Typing in Virtual Reality with a Force-Based Digital Thimble. In Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems, CHI EA '22. Association for Computing Machinery, New York, NY, USA, 2022. 3.
[10]
J. J. Dudley, H. Benko, D. Wigdor, and P. O. Kristensson. Performance Envelopes of Virtual Keyboard Text Input Strategies in Virtual Reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 289–300, Oct. 2019. 3, 4.
[11]
J. J. Dudley, K. Vertanen, and P. O. Kristensson. Fast and Precise Touch-Based Text Entry for Head-Mounted Augmented Reality with Variable Occlusion. ACM Transactions on Computer-Human Interaction, 25 (6): pp. 30:1–30:40, Dec. 2018. 1,2, 3.
[12]
A. Fowler, K. Partridge, C. Chelba, X. Bi, T. Ouyang, and S. Zhai. Effects of Language Modeling and Its Personalization on Touchscreen Typing Performance. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI'15, pp. 649–658. Association for Computing Machinery, New York, NY, USA, 2015. 9.
[13]
C. R. Foy, J. J. Dudley, A. Gupta, H. Benko, and P. O. Kristensson. Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI'21. Association for Computing Machinery, New York, NY, USA, 2021. 4.
[14]
M. Frutos-Pascual, C. Gale, J. M. Harrison, C. Creed, and I. Williams. “Character Input in Augmented Reality: An Evaluation of Keyboard Position and Interaction Visualisation for Head-Mounted Displays”. In C. Ardito, R. Lanzilotti, A. Malizia, H. Petrie, A. Piccinno, G. Desolda, and K. Inkpen, eds., Human-Computer Interaction - INTERACT 2021, pp. 480–501. Cham: Springer International Publishing, 2021. 3.
[15]
D. Gaines, J. Dudley, P. O. Kristensson, and K. Vertanen. “Statistical Keyboard Decoding”. In J. H. Williamson, A. Oulasvirta, P. O. Kristensson, and N. Banovic, eds., Bayesian Methods for Interaction and Design, pp. 188–211. Cambridge University Press, 2022. 4.
[16]
D. Gaines, P.O. Kristensson, and K. Vertanen. Enhancing the Composition Task in Text Entry Studies: Eliciting Difficult Text and Improving Error Rate Calculation. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI '21. Association for Computing Machinery, New York, NY, USA, 2021. 9.
[17]
J. Goodman, G. Venolia, K. Steury, and C. Parker. Language Modeling for Soft Keyboards. In Proceedings of the 7th International Conference on Intelligent User Interfaces, IUI '02, pp. 194–195. Association for Computing Machinery, New York, NY, USA, 2002. 1.
[18]
J. Grubert, L. Witzani, E. Ofek, M. Pahud, M. Kranz, and P. O. Kristensson. Text Entry in Immersive Head-Mounted Display-Based Virtual Reality Using Standard Keyboards. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pp. 159–166, Mar. 2018. 2.
[19]
A. Gupta, C. Ji, H.-S. Yeo, A. Quigley, and D. Vogel. RotoSwype: Word-Gesture Typing Using a Ring. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI'19, pp. 1–12. Association for Computing Machinery, New York, NY, USA, 2019. 2, 3.
[20]
A. Gupta, M. Samad, K. Kin, P. O. Kristensson, and H. Benko. Investigating Remote Tactile Feedback for Mid-Air Text-Entry in Virtual Reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 350–360, Nov. 2020. ISSN:. 4.
[21]
S. Han, B. Liu, R. Cabezas, C. D. Twigg, P. Zhang, J. Petkau, T.-H. Yu, C.-J. Tai, M. Akbay, Z. Wang, A. Nitzan, G. Dong, Y. Ye, L. Tao, C. Wan, and R. Wang. “MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality”. in ACM Trans. Graph., 39 (4), July 2020. Place: New York, NY, USA Publisher: Association for Computing Machinery. 1.
[22]
J. Henderson, J. Ceha, and E. Lank. STAT: Subtle Typing Around the Thigh for Head-Mounted Displays. In 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services, Mobile-HCI '20. Association for Computing Machinery, New York, NY, USA, 2020. 3.
[23]
F. Kern, F. Niebling, and M. E. Latoschik. Text input for non-stationary xr workspaces: Investigating tap and word-gesture keyboards in virtual and augmented reality. IEEE Transactions on Visualization and Computer Graphics, 29 (5): pp. 2658–2669, 2023. 2, 3, 4.
[24]
P. O. Kristensson. Discrete and Continuous Shape Writing for Text Entry and Control. PhD Thesis, Institutionen for datavetenskap, 2007. 2.
[25]
P. O. Kristensson. Five Challenges for Intelligent Text Entry Methods. AI Magazine, 30 (4): pp. 85–83, 2009. 2.
[26]
P. O. Kristensson. Next-Generation Text Entry. in Computer, 48 (7): pp. 84–87, July 2015. Conference Name: Computer. 2, 8.
[27]
P. O. Kristensson. “Statistical Language Processing for Text Entry”. In A. Oulasvirta, P. O. Kristensson, X. Bi, and A. Howes, eds., Computational Interaction. Oxford, UK: Oxford University Press, 2018. 4.
[28]
P. O. Kristensson and K. Vertanen. Performance Comparisons of Phrase Sets and Presentation Styles for Text Entry Evaluations. In Proceedings of the 2012 ACM International Conference on Intelligent User Interfaces, IUI'12, pp. 29–32. Association for Computing Machinery, New York, NY, USA, 2012. 9.
[29]
P. O. Kristensson and K. Vertanen. The Inviscid Text Entry Rate and Its Application as a Grand Goal for Mobile Text Entry. In Proceedings of the 16th International Conference on Human-Computer Interaction with Mobile Devices & Services, MobileHCI'14, pp. 335–338. Association for Computing Machinery, New York, NY, USA, 2014. 9.
[30]
P.-O. Kristensson and S. Zhai. SHARK2: A Large Vocabulary Shorthand Writing System for Pen-Based Computers. In Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology, UIST '04, pp. 43–52. Association for Computing Machinery, New York, NY, USA, 2004., 2, 6.
[31]
P.-O. Kristensson and S. Zhai. Relaxing Stylus Typing Precision by Geometric Pattern Matching. In Proceedings of the 10th International Conference on Intelligent User Interfaces, IUI'05, pp. 151–158. Association for Computing Machinery, New York, NY, USA, 2005. 1.
[32]
Y. Lee and G. J. Kim. “Vitty: Virtual Touch Typing Interface with Added Finger Buttons”. In S. Lackey and J. Chen, eds., Virtual, Augmented and Mixed Reality, pp. 111–119. Cham: Springer International Publishing, 2017. 3.
[33]
L. A. Leiva, S. Kim, W. Cui, X. Bi, and A. Oulasvirta. How we swipe: A large-scale shape-writing dataset and empirical findings. In Proceedings of the 23rd International Conference on Mobile Human-Computer Interaction, MobileHCI '21. Association for Computing Machinery, New York, NY, USA, 2021. 2, 3, 7.
[34]
X. Lu, D. Yu, H.-N. Liang, and J. Goncalves. IText: Hands-Free Text Entry on an Imaginary Keyboard for Augmented Reality Systems. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST '21, pp. 815–825. Association for Computing Machinery, New York, NY, USA, 2021. 3.
[35]
I. S. MacKenzie and R. W. Soukoreff. Phrase Sets for Evaluating Text Entry Techniques. In CHI ‘03 Extended Abstracts on Human Factors in Computing Systems, CHI EA’03, pp. 754–755. Association for Computing Machinery, New York, NY, USA, 2003. 7.
[36]
A. Markussen, M. R. Jakobsen, and K. Hornbaek. Vulture: A Mid-air Word-gesture Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI'14, pp. 1073–1082. ACM, New York, NY, USA, 2014. 2, 3,5, 9.
[37]
A. K. Mutasim, A. U. Batmaz, M. Hudhud Mughrabi, and W. Stuerzlinger. Does Repeatedly Typing the Same Phrase Provide a Good Estimate of Expert Text Entry Performance? In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, CHI EA '23. Association for Computing Machinery, New York, NY, USA, 2023. 6.
[38]
T. Ouyang, D. Rybach, F. Beaufays, and M. Riley. Mobile Keyboard Input Decoding with Finite-State Transducers. CoRR, abs/1704.03987, 2017. _eprint: 1704.03987. 4.
[39]
T. Paek and B.-J. P. Hsu. Sampling Representative Phrase Sets for Text Entry Experiments: A Procedure and Public Resource. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '11, pp. 2477–2480. Association for Computing Machinery, New York, NY, USA, 2011. 9.
[40]
S. Reyal, S. Zhai, and P. O. Kristensson. Performance and User Experience of Touchscreen and Gesture Keyboards in a Lab Setting and in the Wild. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI'15, pp. 679–688. Association for Computing Machinery, New York, NY, USA, 2015. 2. 6.
[41]
M. Richardson, M. Durasoff, and R. Wang. Decoding Surface Touch Typing from Hand-Tracking. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, UIST '20, pp. 686–696. Association for Computing Machinery, New York, NY, USA, Oct. 2020. 3.
[42]
J. Shen, J. Dudley, and P. O. Kristensson. Simulating Realistic Human Motion Trajectories of Mid-Air Gesture Typing. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 393–402, 2021. 2.
[43]
J. Shen, J. Dudley, and P. O. Kristensson. Fast and Robust Mid-Air Gesture Typing for AR Headsets Using 3D Trajectory Decoding. IEEE Transactions on Visualization and Computer Graphics. 2023. 2.
[44]
J. Shen, J. Hu, J. J. Dudley, and P. O. Kristensson. Personalization of a Mid-Air Gesture Keyboard using Multi-Objective Bayesian Optimization. In 2022 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 702–710, 2022. 2.
[45]
M. Speicher, A. M. Feit, P. Ziegler, and A. Krüger. Selection-based text entry in virtual reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI'18, pp. 1–13. Association for Computing Machinery, New York, NY, USA, 2018. 3.
[46]
F. Spiess, P. Weber, and H. Schuldt. Direct Interaction Word-Gesture Text Input in Virtual Reality. In 2022 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 140–143, 2022. 3.
[47]
K. Vertanen, D. Gaines, C. Fletcher, A. M. Stanage, R. Watling, and P. O. Kristensson. Velociwatch: Designing and evaluating a virtual keyboard for the input of challenging text. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI'19, pp. 1–14. Association for Computing Machinery, New York, NY, USA, 2019. 9.
[48]
K. Vertanen and P. Kristensson. “Mining, analyzing, and modeling text written on mobile devices”. in Natural Language Engineering, 27 (1): pp. 1–33, 2021. Publisher: Cambridge University Press. 4.
[49]
K. Vertanen and P. O. Kristensson. A Versatile Dataset for Text Entry Evaluations Based on Genuine Mobile Emails. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, MobileHCI'11, pp. 295–298. Association for Computing Machinery, New York, NY, USA, Aug. 2011. 5, 6.
[50]
K. Vertanen and P. O. Kristensson. Complementing Text Entry Evaluations with a Composition Task. ACM Trans. Comput.-Hum. Interact., 21 (2), feb 2014. 9.
[51]
K. Vertanen, H. Memmi, J. Emge, S. Reyal, and P. O. Kristensson. VelociTap: Investigating Fast Mobile Text Entry Using Sentence-Based Decoding of Touchscreen Keyboard Input. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, CHI'15, pp. 659–668. Association for Computing Machinery, New York, NY, USA, 2015. 1, 4.
[52]
J. Walker, B. Li, K. Vertanen, and S. Kuhl. Efficient Typing on a Visually Occluded Physical Keyboard. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI'17, pp. 5457–5461. ACM, New York, NY, USA, 2017. 2.
[53]
Y. Wang, Y. Wang, J. Chen, Y. Wang, J. Yang, T. Jiang, and J. He. “Investigating the Performance of Gesture-Based Input for Mid-Air Text Entry in a Virtual Environment: A Comparison of Hand-Up versus Hand-Down Postures”. in Sensors, 21 (5): p. 1582, Jan. 2021. Number: 5 Publisher: Multidisciplinary Digital Publishing Institute. 2, 3.
[54]
W. Xu, H.-N. Liang, A. He, and Z. Wang. Pointing and Selection Methods for Text Entry in Augmented Reality Head Mounted Displays. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), pp. 279–288, 2019. 3.
[55]
N. Yanagihara, B. Shizuki, and S. Takahashi. Text Entry Method for Immersive Virtual Environments Using Curved Keyboard. In 25th ACM Symposium on Virtual Reality Software and Technology, VRST '19, pp. 1–2. Association for Computing Machinery, New York, NY, USA, Nov. 2019. 2, 3.
[56]
X. Yi, C. Yu, M. Zhang, S. Gao, K. Sun, and Y. Shi. ATK: Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data. In Proceedings of the 28th Annual ACM Symposium on User Interface Software and Technology, UIST '15, pp. 539–548. Association for Computing Machinery, New York, NY, USA, 2015. 4.
[57]
S. J. Young, N. H. Russell, and J. H. S. Thornton. Token Passing: a Simple Conceptual Model for Connected Recognition Systems. 1989. Publisher: University of Cambridge, Dept. of Engineering. 4.
[58]
C. Yu, Y. Gu, Z. Yang, X. Yi, H. Luo, and Y. Shi. Tap, Dwell or Gesture?: Exploring Head-Based Text Entry Techniques for HMDs. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI '17, pp. 4479–4488. ACM, New York, NY, USA, 2017. 2, 3, 5, 9.
[59]
S. Zhai and P.-O. Kristensson. Shorthand Writing on Stylus Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI '03, pp. 97–104. Association for Computing Machinery, New York, NY, USA, 2003. 1, 2.
[60]
S. Zhai and P. O. Kristensson. The Word-gesture Keyboard: Reimagining Keyboard Interaction. Communications of the ACM, 55 (9): pp. 91–101, Sept. 2012. 2.
[61]
S. Zhai, P. O. Kristensson, P. Gong, M. Greiner, S. A. Peng, L. M. Liu, and A. Dunnigan. Shapewriter on the Iphone: From the Laboratory to the Real World. In CHI ‘09 Extended Abstracts on Human Factors in Computing Systems, CHI EA’09, pp. 2667–2670. Association for Computing Machinery, New York, NY, USA, 2009. 2.

Cited By

View all
  • (2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
  • (2024)StegoType: Surface Typing from Egocentric CamerasAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686762(1-14)Online publication date: 13-Oct-2024
  • (2024)StegoType: Surface Typing from Egocentric CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676343(1-14)Online publication date: 13-Oct-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics
IEEE Transactions on Visualization and Computer Graphics  Volume 29, Issue 11
Nov. 2023
465 pages

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 04 October 2023

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 13 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2024)OnArmQWERTY: An Empirical Evaluation of On-Arm Tap Typing for AR HMDsProceedings of the 2024 ACM Symposium on Spatial User Interaction10.1145/3677386.3682084(1-12)Online publication date: 7-Oct-2024
  • (2024)StegoType: Surface Typing from Egocentric CamerasAdjunct Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3672539.3686762(1-14)Online publication date: 13-Oct-2024
  • (2024)StegoType: Surface Typing from Egocentric CamerasProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676343(1-14)Online publication date: 13-Oct-2024
  • (2024)TouchInsight: Uncertainty-aware Rapid Touch and Text Input for Mixed Reality from Egocentric VisionProceedings of the 37th Annual ACM Symposium on User Interface Software and Technology10.1145/3654777.3676330(1-16)Online publication date: 13-Oct-2024

View Options

View options

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media