Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article
Open access

From 2D to 3D: Facilitating Single-Finger Mid-Air Typing on QWERTY Keyboards with Probabilistic Touch Modeling

Published: 28 March 2023 Publication History

Abstract

Mid-air text entry on virtual keyboards suffers from the lack of tactile feedback, which brings challenges to both tap detection and input prediction. In this paper, we explored the feasibility of single-finger typing on virtual QWERTY keyboards in mid-air. We first conducted a study to examine users' 3D typing behavior on different sizes of virtual keyboards. Results showed that the participants perceived the vertical projection of the lowest point on the keyboard during a tap as the target location and inferring taps based on the intersection between the finger and the keyboard was not applicable. Aiming at this challenge, we derived a novel input prediction algorithm that took the uncertainty in tap detection into a calculation as probability, and performed probabilistic decoding that could tolerate false detection. We analyzed the performance of the algorithm through a full-factorial simulation. Results showed that the SVM-based probabilistic touch detection together with a 2D elastic probabilistic decoding algorithm (elasticity = 2) could achieve the optimal top-5 accuracy of 94.2%. In the evaluation user study, the participants reached a single-finger typing speed of 26.1 WPM with 3.2% uncorrected word-level error rate, which was significantly better than both tap-based and gesture-based baseline techniques. Also, the proposed technique received the highest preference score from the users, proving its usability in real text entry tasks.

References

[1]
Diar Abdlkarim, Massimiliano Di Luca, Poppy Aves, Sang-Hoon Yeo, R. Chris Miall, Peter Holland, and Joseph M. Galea. 2022. A Methodological Framework to Assess the Accuracy of Virtual Reality Hand-Tracking Systems: A case study with the Oculus Quest 2. bioRxiv (2022). https://doi.org/10.1101/2022.02.18.481001 arXiv:https://www.biorxiv.org/content/early/2022/02/20/2022.02.18.481001.full.pdf
[2]
Jiban Adhikary and Keith Vertanen. 2021. Typing on Midair Virtual Keyboards: Exploring Visual Designs and Interaction Styles. In IFIP Conference on Human-Computer Interaction. Springer, 132--151.
[3]
Christoph Amma, Marcus Georgi, and Tanja Schultz. 2012. Airwriting: Hands-Free Mobile Text Input by Spotting and Continuous Recognition of 3d-Space Handwriting with Inertial Sensors. In 2012 16th International Symposium on Wearable Computers. 52--59. https://doi.org/10.1109/ISWC.2012.21
[4]
Shiri Azenkot and Shumin Zhai. 2012. Touch Behavior with Different Postures on Soft Smartphone Keyboards. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (San Francisco, California, USA) (MobileHCI '12). Association for Computing Machinery, New York, NY, USA, 251--260. https://doi.org/10.1145/2371574.2371612
[5]
Sebastian Boring, David Ledo, Xiang 'Anthony' Chen, Nicolai Marquardt, Anthony Tang, and Saul Greenberg. 2012. The Fat Thumb: Using the Thumb's Contact Size for Single-Handed Mobile Interaction. In Proceedings of the 14th International Conference on Human-Computer Interaction with Mobile Devices and Services (San Francisco, California, USA) (MobileHCI '12). Association for Computing Machinery, New York, NY, USA, 39--48. https://doi.org/10.1145/2371574.2371582
[6]
Thorsten Brants and Alex Franz. 2006. Web 1T 5-gram Ver. 1.
[7]
John Dudley, Hrvoje Benko, Daniel Wigdor, and Per Ola Kristensson. 2019. Performance envelopes of virtual keyboard text input strategies in virtual reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 289--300.
[8]
John J. Dudley, Keith Vertanen, and Per Ola Kristensson. 2018. Fast and Precise Touch-Based Text Entry for Head-Mounted Augmented Reality with Variable Occlusion. ACM Trans. Comput.-Hum. Interact. 25, 6, Article 30 (Dec. 2018), 40 pages. https://doi.org/10.1145/3232163
[9]
Leah Findlater and Jacob Wobbrock. 2012. Personalized Input: Improving Ten-Finger Touchscreen Typing through Automatic Adaptation. Association for Computing Machinery, New York, NY, USA, 815--824. https://doi.org/10.1145/2207676.2208520
[10]
Leah Findlater, Jacob O. Wobbrock, and Daniel Wigdor. 2011. Typing on Flat Glass: Examining Ten-Finger Expert Typing Patterns on Touch Surfaces. Association for Computing Machinery, New York, NY, USA, 2453--2462. https://doi.org/10.1145/1978942.1979301
[11]
Conor R. Foy, John J. Dudley, Aakar Gupta, Hrvoje Benko, and Per Ola Kristensson. 2021. Understanding, Detecting and Mitigating the Effects of Coactivations in Ten-Finger Mid-Air Typing in Virtual Reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (Yokohama, Japan) (CHI '21). Association for Computing Machinery, New York, NY, USA, Article 287, 11 pages. https://doi.org/10.1145/3411764.3445671
[12]
Mayank Goel, Leah Findlater, and Jacob Wobbrock. 2012. WalkType: Using Accelerometer Data to Accomodate Situational Impairments in Mobile Touch Screen Text Entry. Association for Computing Machinery, New York, NY, USA, 2687--2696. https://doi.org/10.1145/2207676.2208662
[13]
Mayank Goel, Alex Jansen, Travis Mandel, Shwetak N. Patel, and Jacob O. Wobbrock. 2013. ContextType: Using Hand Posture Information to Improve Mobile Touch Screen Text Entry. Association for Computing Machinery, New York, NY, USA, 2795--2798. https://doi.org/10.1145/2470654.2481386
[14]
Joshua Goodman, Gina Venolia, Keith Steury, and Chauncey Parker. 2002. Language Modeling for Soft Keyboards. In Proc. IUI'02 (San Francisco, California, USA). Association for Computing Machinery, New York, NY, USA, 194--195. https://doi.org/10.1145/502716.502753
[15]
Shangchen Han, Beibei Liu, Randi Cabezas, Christopher D. Twigg, Peizhao Zhang, Jeff Petkau, Tsz-Ho Yu, Chun-Jung Tai, Muzaffer Akbay, Zheng Wang, Asaf Nitzan, Gang Dong, Yuting Ye, Lingling Tao, Chengde Wan, and Robert Wang. 2020. MEgATrack: Monochrome Egocentric Articulated Hand-Tracking for Virtual Reality. ACM Trans. Graph. 39, 4, Article 87 (aug 2020), 13 pages. https://doi.org/10.1145/3386569.3392452
[16]
Christian Holz and Patrick Baudisch. 2011. Understanding Touch. Association for Computing Machinery, New York, NY, USA, 2501--2510. https://doi.org/10.1145/1978942.1979308
[17]
S. Katz. 1987. Estimation of probabilities from sparse data for the language model component of a speech recognizer. IEEE Transactions on Acoustics, Speech, and Signal Processing 35, 3 (1987), 400--401. https://doi.org/10.1109/TASSP.1987.1165125
[18]
Seoktae Kim, Minjung Sohn, Jinhee Pak, and Woohun Lee. 2006. One-Key Keyboard: A Very QWERTY Keyboard Supporting Text Entry for Wearable Computing. In Proceedings of the 18th Australia Conference on Computer-Human Interaction: Design: Activities, Artefacts and Environments (Sydney, Australia) (OZCHI '06). Association for Computing Machinery, New York, NY, USA, 305--308. https://doi.org/10.1145/1228175.1228229
[19]
Chen Liang, Chi Hsia, Chun Yu, Yukang Yan, Yuntao Wang, and Yuanchun Shi. 2023. DRG-Keyboard: Enabling Subtle Gesture Typing on the Fingertip with Dual IMU Rings. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 6, 4, Article 170 (jan 2023), 30 pages. https://doi.org/10.1145/3569463
[20]
Jia-Wei Lin, Ping-Hsuan Han, Jiun-Yu Lee, Yang-Sheng Chen, Ting-Wei Chang, Kuan-Wen Chen, and Yi-Ping Hung. 2017. Visualizing the Keyboard in Virtual Reality for Enhancing Immersive Experience. In ACM SIGGRAPH 2017 Posters (Los Angeles, California) (SIGGRAPH '17). Association for Computing Machinery, New York, NY, USA, Article 35, 2 pages. https://doi.org/10.1145/3102163.3102175
[21]
Yiqin Lu, Chun Yu, Xin Yi, Yuanchun Shi, and Shengdong Zhao. 2017. BlindType: Eyes-Free Text Entry on Handheld Touchpad by Leveraging Thumb's Muscle Memory. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 2, Article 18 (June 2017), 24 pages. https://doi.org/10.1145/3090083
[22]
I. S. Mackenzie. 2015, http://www.yorku.ca/mack/RN-TextEntrySpeed.html. A Note on Calculating Text Entry Speed.
[23]
I. Scott MacKenzie and R. William Soukoreff. 2003. Phrase Sets for Evaluating Text Entry Techniques. Association for Computing Machinery, New York, NY, USA, 754--755. https://doi.org/10.1145/765891.765971
[24]
C. Macleod, N. Ide, and R. Grishman. 2002. The American National Corpus: A Standardized Resource for American English. (2002).
[25]
Anders Markussen, Mikkel Rønne Jakobsen, and Kasper Hornbæk. 2014. Vulture: A Mid-Air Word-Gesture Keyboard. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 1073--1082. https://doi.org/10.1145/2556288.2556964
[26]
I. S. P. Nation and R. Waring. 1997. Vocabulary size, text coverage, and word lists. n.schmitt m.mccarthy vocabulary description acquisition pedagogy (1997).
[27]
Tao Ni, Doug Bowman, and Chris North. 2011. AirStroke: Bringing Unistroke Text Entry to Freehand Gesture Interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Vancouver, BC, Canada) (CHI '11). Association for Computing Machinery, New York, NY, USA, 2473--2476. https://doi.org/10.1145/1978942.1979303
[28]
Duc-Minh Pham and Wolfgang Stuerzlinger. 2019. HawKEY: Efficient and Versatile Text Entry for Virtual Reality. In 25th ACM Symposium on Virtual Reality Software and Technology (Parramatta, NSW, Australia) (VRST '19). Association for Computing Machinery, New York, NY, USA, Article 21, 11 pages. https://doi.org/10.1145/3359996.3364265
[29]
J Platt. 1999. Probabilistic outputs for SVMs and comparisons to regularized likehood methods. In Advances in Large Margin Classifiers. MIT Press.
[30]
Daniel R. Rashid and Noah A. Smith. 2008. Relative Keyboard Input System. In Proceedings of the 13th International Conference on Intelligent User Interfaces (Gran Canaria, Spain) (IUI '08). Association for Computing Machinery, New York, NY, USA, 397--400. https://doi.org/10.1145/1378773.1378839
[31]
Shyam Reyal, Shumin Zhai, and Per Ola Kristensson. 2015. Performance and User Experience of Touchscreen and Gesture Keyboards in a Lab Setting and in the Wild. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (Seoul, Republic of Korea) (CHI '15). Association for Computing Machinery, New York, NY, USA, 679--688. https://doi.org/10.1145/2702123.2702597
[32]
Mark Richardson, Matt Durasoff, and Robert Wang. 2020. Decoding Surface Touch Typing from Hand-Tracking. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST '20). Association for Computing Machinery, New York, NY, USA, 686--696. https://doi.org/10.1145/3379337.3415816
[33]
Mark Richardson, Matt Durasoff, and Robert Wang. 2020. Decoding Surface Touch Typing from Hand-Tracking. Association for Computing Machinery, New York, NY, USA, 686--696. https://doi.org/10.1145/3379337.3415816
[34]
Weinan Shi, Chun Yu, Shuyi Fan, Feng Wang, Tong Wang, Xin Yi, Xiaojun Bi, and Yuanchun Shi. 2019. VIPBoard: Improving Screen-Reader Keyboard for Visually Impaired People with Character-Level Auto Correction. Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3290605.3300747
[35]
Weinan Shi, Chun Yu, Xin Yi, Zhen Li, and Yuanchun Shi. 2018. TOAST: Ten-Finger Eyes-Free Typing on Touchable Surfaces. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 2, 1, Article 33 (March 2018), 23 pages. https://doi.org/10.1145/3191765
[36]
R. William Soukoreff and I. Scott MacKenzie. 2003. Metrics for Text Entry Research: An Evaluation of MSD and KSPC, and a New Unified Error Metric. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, Florida, USA) (CHI '03). Association for Computing Machinery, New York, NY, USA, 113--120. https://doi.org/10.1145/642611.642632
[37]
Marco Speicher, Anna Maria Feit, Pascal Ziegler, and Antonio Krüger. 2018. Selection-Based Text Entry in Virtual Reality. Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3173574.3174221
[38]
Keith Vertanen, Crystal Fletcher, Dylan Gaines, Jacob Gould, and Per Ola Kristensson. 2018. The Impact of Word, Multiple Word, and Sentence Input on Virtual Keyboard Decoding Performance. Association for Computing Machinery, New York, NY, USA, 1--12. https://doi.org/10.1145/3173574.3174200
[39]
Keith Vertanen, Dylan Gaines, Crystal Fletcher, Alex M. Stanage, Robbie Watling, and Per Ola Kristensson. 2019. VelociWatch: Designing and Evaluating a Virtual Keyboard for the Input of Challenging Text. Association for Computing Machinery, New York, NY, USA, 1--14. https://doi.org/10.1145/3290605.3300821
[40]
Keith Vertanen, Haythem Memmi, Justin Emge, Shyam Reyal, and Per Ola Kristensson. 2015. VelociTap: Investigating Fast Mobile Text Entry Using Sentence-Based Decoding of Touchscreen Keyboard Input. Association for Computing Machinery, New York, NY, USA, 659--668. https://doi.org/10.1145/2702123.2702135
[41]
James Walker, Bochao Li, Keith Vertanen, and Scott Kuhl. 2017. Efficient Typing on a Visually Occluded Physical Keyboard. Association for Computing Machinery, New York, NY, USA, 5457--5461. https://doi.org/10.1145/3025453.3025783
[42]
Yuntao Wang, Ao Yu, Xin Yi, Yuanwei Zhang, Ishan Chatterjee, Shwetak Patel, and Yuanchun Shi. 2021. Facilitating Text Entry on Smartphones with QWERTY Keyboard for Users with Parkinson's Disease. Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3411764.3445352
[43]
Daryl Weir, Henning Pohl, Simon Rogers, Keith Vertanen, and Per Ola Kristensson. 2014. Uncertain Text Entry on Mobile Devices. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Toronto, Ontario, Canada) (CHI '14). Association for Computing Machinery, New York, NY, USA, 2307--2316. https://doi.org/10.1145/2556288.2557412
[44]
Andrew D. Wilson and Maneesh Agrawala. 2006. Text Entry Using a Dual Joystick Game Controller. Association for Computing Machinery, New York, NY, USA, 475--478. https://doi.org/10.1145/1124772.1124844
[45]
Zhican Yang, Chun Yu, Xin Yi, and Yuanchun Shi. 2019. Investigating Gesture Typing for Indirect Touch. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 3, Article 117 (Sept. 2019), 22 pages. https://doi.org/10.1145/3351275
[46]
Xin Yi, Chen Wang, Xiaojun Bi, and Yuanchun Shi. 2020. PalmBoard: Leveraging Implicit Touch Pressure in Statistical Decoding for Indirect Text Entry. Association for Computing Machinery, New York, NY, USA, 1--13. https://doi.org/10.1145/3313831.3376441
[47]
Xin Yi, Chun Yu, Weinan Shi, and Yuanchun Shi. 2017. Is it too small?: Investigating the performances and preferences of users when typing on tiny QWERTY keyboards. International Journal of Human-Computer Studies 106 (2017), 44--62. https://doi.org/10.1016/j.ijhcs.2017.05.001
[48]
Xin Yi, Chun Yu, Mingrui Zhang, Sida Gao, Ke Sun, and Yuanchun Shi. 2015. ATK: Enabling Ten-Finger Freehand Typing in Air Based on 3D Hand Tracking Data. In Proceedings of the 28th Annual ACM Symposium on User Interface Software amp; Technology (Charlotte, NC, USA) (UIST '15). Association for Computing Machinery, New York, NY, USA, 539--548. https://doi.org/10.1145/2807442.2807504
[49]
Chun Yu, Yizheng Gu, Zhican Yang, Xin Yi, Hengliang Luo, and Yuanchun Shi. 2017. Tap, Dwell or Gesture? Exploring Head-Based Text Entry Techniques for HMDs. Association for Computing Machinery, New York, NY, USA, 4479--4488. https://doi.org/10.1145/3025453.3025964

Cited By

View all
  • (2024)CrossKeys: Text Entry for Virtual Reality Using a Single Controller via Wrist RotationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2358456(1-13)Online publication date: 5-Jun-2024
  • (2024)Research on pedestrian counting based on millimeter waveCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-023-00145-66:1(82-100)Online publication date: 18-Feb-2024
  • (2024)Dynamic Liveness Detection Based on Fusion of mmWave Radar and VisionWireless Sensor Networks10.1007/978-981-97-1010-2_4(42-55)Online publication date: 1-Mar-2024
  • Show More Cited By

Index Terms

  1. From 2D to 3D: Facilitating Single-Finger Mid-Air Typing on QWERTY Keyboards with Probabilistic Touch Modeling

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 7, Issue 1
      March 2023
      1243 pages
      EISSN:2474-9567
      DOI:10.1145/3589760
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 28 March 2023
      Published in IMWUT Volume 7, Issue 1

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. text entry
      2. virtual reality

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • the grant from the Institute for Guo Qiang, Tsinghua University
      • Natural Science Foundation of China (NSFC)

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)494
      • Downloads (Last 6 weeks)57
      Reflects downloads up to 21 Sep 2024

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)CrossKeys: Text Entry for Virtual Reality Using a Single Controller via Wrist RotationInternational Journal of Human–Computer Interaction10.1080/10447318.2024.2358456(1-13)Online publication date: 5-Jun-2024
      • (2024)Research on pedestrian counting based on millimeter waveCCF Transactions on Pervasive Computing and Interaction10.1007/s42486-023-00145-66:1(82-100)Online publication date: 18-Feb-2024
      • (2024)Dynamic Liveness Detection Based on Fusion of mmWave Radar and VisionWireless Sensor Networks10.1007/978-981-97-1010-2_4(42-55)Online publication date: 1-Mar-2024
      • (2023)STAR: Smartphone-analogous Typing in Augmented RealityProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606803(1-13)Online publication date: 29-Oct-2023
      • (2023)ShadowTouch: Enabling Free-Form Touch-Based Hand-to-Surface Interaction with Wrist-Mounted Illuminant by Shadow ProjectionProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606785(1-14)Online publication date: 29-Oct-2023
      • (2023)Fast and Robust Mid-Air Gesture Typing for AR Headsets using 3D Trajectory DecodingIEEE Transactions on Visualization and Computer Graphics10.1109/TVCG.2023.332021829:11(4622-4632)Online publication date: 2-Oct-2023

      View Options

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Get Access

      Login options

      Full Access

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media