Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

IndexPen: Two-Finger Text Input with Millimeter-Wave Radar

Published: 07 July 2022 Publication History

Abstract

In this paper, we introduce IndexPen, a novel interaction technique for text input through two-finger in-air micro-gestures, enabling touch-free, effortless, tracking-based interaction, designed to mirror real-world writing. Our system is based on millimeter-wave radar sensing, and does not require instrumentation on the user. IndexPen can successfully identify 30 distinct gestures, representing the letters A-Z, as well as Space, Backspace, Enter, and a special Activation gesture to prevent unintentional input. Additionally, we include a noise class to differentiate gesture and non-gesture noise. We present our system design, including the radio frequency (RF) processing pipeline, classification model, and real-time detection algorithms. We further demonstrate our proof-of-concept system with data collected over ten days with five participants yielding 95.89% cross-validation accuracy on 31 classes (including noise). Moreover, we explore the learnability and adaptability of our system for real-world text input with 16 participants who are first-time users to IndexPen over five sessions. After each session, the pre-trained model from the previous five-user study is calibrated on the data collected so far for a new user through transfer learning. The F-1 score showed an average increase of 9.14% per session with the calibration, reaching an average of 88.3% on the last session across the 16 users. Meanwhile, we show that the users can type sentences with IndexPen at 86.2% accuracy, measured by string similarity. This work builds a foundation and vision for future interaction interfaces that could be enabled with this paradigm.

Supplementary Material

wei (wei.zip)
Supplemental movie, appendix, image and software files for, IndexPen: Two-Finger Text Input with Millimeter-Wave Radar

References

[1]
Heba Abdelnasser, Moustafa Youssef, and Khaled A Harras. 2015. Wigest: A ubiquitous wifi-based gesture recognition system. In 2015 IEEE Conference on Computer Communications (INFOCOM). IEEE, 1472--1480.
[2]
Ibrahim Alnujaim, Hashim Alali, Faisal Khan, and Youngwook Kim. 2018. Hand gesture recognition using input impedance variation of two antennas with transfer learning. IEEE Sensors Journal 18, 10 (2018), 4129--4135.
[3]
Amin Arbabian, Steven Callender, Shinwon Kang, Mustafa Rangwala, and Ali M Niknejad. 2013. A 94 GHz mm-wave-to-baseband pulsed-radar transceiver with applications in imaging and gesture recognition. IEEE Journal of Solid-State Circuits 48, 4 (2013), 1055--1071.
[4]
Maryam Asadi-Aghbolaghi, Albert Clapes, Marco Bellantonio, Hugo Jair Escalante, Víctor Ponce-López, Xavier Baró, Isabelle Guyon, Shohreh Kasaei, and Sergio Escalera. 2017. A survey on deep learning based approaches for action and gesture recognition in image sequences. In 2017 12th IEEE international conference on automatic face & gesture recognition (FG 2017). IEEE, 476--483.
[5]
Sebastian Bock and Martin Weiß. 2019. A proof of local convergence for the Adam optimizer. In 2019 International Joint Conference on Neural Networks (IJCNN). IEEE, 1--8.
[6]
Fazli Can and Jon M Patton. 2004. Change of writing style with time. Computers and the Humanities 38, 1 (2004), 61--82.
[7]
Edwin Chan, Teddy Seyed, Wolfgang Stuerzlinger, Xing-Dong Yang, and Frank Maurer. 2016. User Elicitation on Single-hand Microgestures. 3403--3414. https://doi.org/10.1145/2858036.2858589
[8]
Liwei Chan, Rong-Hao Liang, Ming-Chang Tsai, Kai-Yin Cheng, Chao-Huai Su, Mike Y Chen, Wen-Huang Cheng, and Bing-Yu Chen. 2013. FingerPad: private and subtle interaction using fingertips. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 255--260.
[9]
Ke-Yu Chen, Shwetak N Patel, and Sean Keller. 2016. Finexus: Tracking precise motions of multiple fingertips using magnetic sensing. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1504--1514.
[10]
Dan C Cireşan, Ueli Meier, and Jürgen Schmidhuber. 2012. Transfer learning for Latin and Chinese characters with deep neural networks. In The 2012 international joint conference on neural networks (IJCNN). IEEE, 1--6.
[11]
David A Cook. 2014. The value of online learning and MRI: finding a niche for expensive technologies. Medical teacher 36, 11 (2014), 965--972.
[12]
Ulysse Côté-Allard, Cheikh Latyr Fall, Alexandre Drouin, Alexandre Campeau-Lecours, Clément Gosselin, Kyrre Glette, François Laviolette, and Benoit Gosselin. 2019. Deep learning for electromyographic hand gesture signal classification using transfer learning. IEEE Transactions on Neural Systems and Rehabilitation Engineering 27, 4 (2019), 760--771.
[13]
Artem Dementyev and Joseph A Paradiso. 2014. WristFlex: low-power gesture input with wrist-worn pressure sensors. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 161--166.
[14]
Fatemeh Fahimi, Zhuo Zhang, Wooi Boon Goh, Tih-Shi Lee, Kai Keng Ang, and Cuntai Guan. 2019. Inter-subject transfer learning with an end-to-end deep convolutional neural network for EEG-based BCI. Journal of neural engineering 16, 2 (2019), 026007.
[15]
Yun Fu and Thomas S Huang. 2007. hMouse: Head tracking driven virtual computer mouse. In 2007 IEEE Workshop on Applications of Computer Vision (WACV'07). IEEE, 30--30.
[16]
Jun Gong, Yang Zhang, Xia Zhou, and Xing-Dong Yang. 2017. Pyro: Thumb-tip gesture recognition using pyroelectric infrared sensing. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. ACM, 553--563.
[17]
Emma Gregory, Barbara Landau, and Michael McCloskey. 2011. Representation of object orientation in children: Evidence from mirror-image confusions. Visual cognition 19, 8 (2011), 1035--1062.
[18]
Eiji Hayashi, Jaime Lien, Nicholas Gillian, Leonardo Giusti, Dave Weber, Jin Yamanaka, Lauren Bedal, and Ivan Poupyrev. 2021. RadarNet: Efficient Gesture Recognition Technique Utilizing a Miniature Radar Sensor. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1--14.
[19]
Wilbert Jan Heeringa. 2004. Measuring dialect pronunciation differences using Levenshtein distance. Ph.D. Dissertation. University Library Groningen][Host].
[20]
Chen-Yu Hsu, Yuchen Liu, Zachary Kabelac, Rumen Hristov, Dina Katabi, and Christine Liu. 2017. Extracting gait velocity and stride length from surrounding radio signals. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. 2116--2126.
[21]
J Stuart Hunter. 1986. The exponentially weighted moving average. Journal of quality technology 18, 4 (1986), 203--210.
[22]
Cesar Iovescu and Sandeep Rao. 2017. The fundamentals of millimeter wave sensors. Texas Instruments, SPYY005 (2017).
[23]
Robert JK Jacob, Audrey Girouard, Leanne M Hirshfield, Michael S Horn, Orit Shaer, Erin Treacy Solovey, and Jamie Zigelbaum. 2008. Reality-based interaction: a framework for post-WIMP interfaces. In Proceedings of the SIGCHI conference on Human factors in computing systems. 201--210.
[24]
Mohita Jaiswal, Vaidehi Sharmay, Abhishek Sharmaz, and Raghuvir Tomar. 2020. Transfer Learning with L2 Norm Regularization for classifying static Two Hand Hindi Sign Language Gestures. In 2020 IEEE 9th International Conference on Communication Systems and Network Technologies (CSNT). IEEE, 44--48.
[25]
Vinay Jayaram, Morteza Alamgir, Yasemin Altun, Bernhard Scholkopf, and Moritz Grosse-Wentrup. 2016. Transfer learning in brain-computer interfaces. IEEE Computational Intelligence Magazine 11, 1 (2016), 20--31.
[26]
Jing Jia, Geng Tu, Xin Deng, Chuchu Zhao, and Wenlong Yi. 2019. Real-time hand gestures system based on leap motion. Concurrency and Computation: Practice and Experience 31, 10 (2019), e4898.
[27]
Faheem Khan, Seong Kyu Leem, and Sung Ho Cho. 2017. Hand-based gesture recognition for vehicular applications using IR-UWB radar. Sensors 17, 4 (2017), 833.
[28]
Eva Kollorz, Jochen Penne, Joachim Hornegger, and Alexander Barke. 2008. Gesture recognition with a time-of-flight camera. International Journal of Intelligent Systems Technologies and Applications 5, 3 (2008), 334.
[29]
Chan F Lam and David Kamins. 1989. Signature recognition through spectral analysis. Pattern Recognition 22, 1 (1989), 39--44.
[30]
Ziheng Li, Zhenyuan Lei, An Yan, Erin Solovey, and Kaveh Pahlavan. 2020. ThuMouse: A micro-gesture cursor input through mmWave radar-based interaction. In 2020 IEEE International Conference on Consumer Electronics (ICCE). IEEE, 1--9.
[31]
Jaime Lien, Nicholas Gillian, M Emre Karagozler, Patrick Amihood, Carsten Schwesig, Erik Olson, Hakim Raja, and Ivan Poupyrev. 2016. Soli: Ubiquitous gesture sensing with millimeter wave radar. ACM Transactions on Graphics (TOG) 35, 4 (2016), 142.
[32]
Haipeng Liu, Yuheng Wang, Anfu Zhou, Hanyue He, Wei Wang, Kunpeng Wang, Peilin Pan, Yixuan Lu, Liang Liu, and Huadong Ma. 2020. Real-time arm gesture recognition in smart home scenarios via millimeter wave sensing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 4, 4 (2020), 1--28.
[33]
Haipeng Liu, Anfu Zhou, Zihe Dong, Yuyang Sun, Jiahe Zhang, Liang Liu, Huadong Ma, Jianhua Liu, and Ning Yang. 2021. M-Gesture: Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar. IEEE Internet of Things Journal (2021), 1--1. https://doi.org/10.1109/JIOT.2021.3098338
[34]
Fabien Lotte and Cuntai Guan. 2010. Regularizing common spatial patterns to improve BCI designs: unified theory and new algorithms. IEEE Transactions on biomedical Engineering 58, 2 (2010), 355--362.
[35]
ZN Low, JH Cheong, and CL Law. 2005. Low-cost PCB antenna for UWB applications. IEEE antennas and wireless propagation letters 4 (2005), 237--239.
[36]
Chris Xiaoxuan Lu, Muhamad Risqi U Saputra, Peijun Zhao, Yasin Almalioglu, Pedro PB de Gusmao, Changhao Chen, Ke Sun, Niki Trigoni, and Andrew Markham. 2020. milliEgo: mmWave Aided Egomotion Estimation with Deep Sensor Fusion. arXiv preprint arXiv:2006.02266 (2020).
[37]
Robin Lu and Yan Zhang. 2013. Balanced debounce circuit with noise filter for digital system. US Patent 8,502,593.
[38]
Yongsen Ma, Gang Zhou, Shuangquan Wang, Hongyang Zhao, and Woosub Jung. 2018. Signfi: Sign language recognition using wifi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 1 (2018), 1--21.
[39]
Giulio Marin, Fabio Dominio, and Pietro Zanuttigh. 2014. Hand gesture recognition with leap motion and kinect devices. In 2014 IEEE International conference on image processing (ICIP). IEEE, 1565--1569.
[40]
Pavlo Molchanov, Shalini Gupta, Kihwan Kim, and Jan Kautz. 2015. Hand gesture recognition with 3D convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition workshops. 1--7.
[41]
Oyebade K Oyedotun and Adnan Khashman. 2017. Deep learning in vision-based static hand gesture recognition. Neural Computing and Applications 28, 12 (2017), 3941--3951.
[42]
Kaveh Pahlavan, Julang Ying, Ziheng Li, Erin Solovey, John Loftus, and Zehua Dong. 2020. RF Cloud for Cyberspace Intelligence. IEEE Access (2020).
[43]
Sameera Palipana, Dariush Salami, Luis A Leiva, and Stephan Sigg. 2021. Pantomime: Mid-air gesture recognition with sparse millimeter-wave radar point clouds. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 1 (2021), 1--27.
[44]
Sinno Jialin Pan and Qiang Yang. 2009. A survey on transfer learning. IEEE Transactions on knowledge and data engineering 22, 10 (2009), 1345--1359.
[45]
Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2013. Whole-home gesture recognition using wireless signals. In Proceedings of the 19th annual international conference on Mobile computing & networking. 27--38.
[46]
Qifan Pu, Sidhant Gupta, Shyamnath Gollakota, and Shwetak Patel. 2015. Gesture recognition using wireless signals. GetMobile: Mobile Computing and Communications 18, 4 (2015), 15--18.
[47]
Edmar Rezende, Guilherme Ruppert, Tiago Carvalho, Fabio Ramos, and Paulo de Geus. 2017. Malicious Software Classification Using Transfer Learning of ResNet-50 Deep Neural Network. In 2017 16th IEEE International Conference on Machine Learning and Applications (ICMLA). 1011--1014. https://doi.org/10.1109/ICMLA.2017.00-19
[48]
Herbert Richter. 2000. Palm pilot holder. US Patent App. 29/114,214.
[49]
Dariush Salami, Ramin Hasibi, Sameera Palipana, Petar Popovski, Tom Michoel, and Stephan Sigg. 2021. Tesla-Rapture: A Lightweight Gesture Recognition System from mmWave Radar Point Clouds. arXiv preprint arXiv:2109.06448 (2021).
[50]
T Scott Saponas, Desney S Tan, Dan Morris, and Ravin Balakrishnan. 2008. Demonstrating the feasibility of using forearm electromyography for muscle-computer interfaces. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 515--524.
[51]
T Scott Saponas, Desney S Tan, Dan Morris, Ravin Balakrishnan, Jim Turner, and James A Landay. 2009. Enabling always-available input with muscle-computer interfaces. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 167--176.
[52]
Yuan-Fu Shao, Masatoshi Chang-Ogimoto, Reinhard Pointner, Yu-Chih Lin, Chen-Ting Wu, and Mike Chen. 2016. SwipeKey: a swipe-based keyboard design for smartwatches. In Proceedings of the 18th International Conference on Human-Computer Interaction with Mobile Devices and Services. 60--71.
[53]
Connor Shorten and Taghi M Khoshgoftaar. 2019. A survey on image data augmentation for deep learning. Journal of Big Data 6, 1 (2019), 1--48.
[54]
Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[55]
Karly A Smith, Clément Csech, David Murdoch, and George Shaker. 2018. Gesture recognition using mm-wave sensor for human-car interface. IEEE sensors letters 2, 2 (2018), 1--4.
[56]
Samuel L Smith, Pieter-Jan Kindermans, Chris Ying, and Quoc V Le. 2017. Don't decay the learning rate, increase the batch size. arXiv preprint arXiv:1711.00489 (2017).
[57]
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research 15, 1 (2014), 1929--1958.
[58]
Srikanth Tammina. 2019. Transfer learning using vgg-16 with deep convolutional neural network for classifying images. International Journal of Scientific and Research Publications (IJSRP) 9, 10 (2019), 143--150.
[59]
Sheng Tan, Linghan Zhang, Zi Wang, and Jie Yang. 2019. MultiTrack: Multi-user tracking and activity recognition using commodity WiFi. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1--12.
[60]
Jilin Tu, Thomas Huang, and Hai Tao. 2005. Face as mouse through visual face tracking. In The 2nd Canadian Conference on Computer and Robot Vision (CRV'05). IEEE, 339--346.
[61]
Saiwen Wang, Jie Song, Jaime Lien, Ivan Poupyrev, and Otmar Hilliges. 2016. Interacting with soli: Exploring fine-grained dynamic gesture recognition in the radio-frequency spectrum. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. ACM, 851--860.
[62]
Yao Wang and Qin-Fan Zhu. 1998. Error control and concealment for video communication: A review. Proc. IEEE 86, 5 (1998), 974--997.
[63]
Frank Weichert, Daniel Bachmann, Bartholomäus Rudak, and Denis Fisseler. 2013. Analysis of the accuracy and robustness of the leap motion controller. Sensors 13, 5 (2013), 6380--6393.
[64]
Xiaoling Xia, Cui Xu, and Bing Nan. 2017. Inception-v3 for flower classification. In 2017 2nd International Conference on Image, Vision and Computing (ICIVC). IEEE, 783--787.
[65]
Xiao-Ling Xia, Cui Xu, and Bing Nan. 2017. Facial expression recognition based on tensorflow platform. In ITM Web of Conferences, Vol. 12. EDP Sciences, 01005.
[66]
Zheer Xu, Pui Chung Wong, Jun Gong, Te-Yen Wu, Aditya Shekhar Nittala, Xiaojun Bi, Jürgen Steimle, Hongbo Fu, Kening Zhu, and Xing-Dong Yang. 2019. TipText: Eyes-Free Text Entry on a Fingertip Keyboard. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 883--899.
[67]
Hui-Shyong Yeo, Gergely Flamich, Patrick Schrempf, David Harris-Birtill, and Aaron Quigley. 2016. Radarcat: Radar categorization for input & interaction. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 833--841.
[68]
Jih-Tsun Yu, Li Yen, and Po-Hsuan Tseng. 2020. mmWave radar-based hand gesture recognition using range-angle image. In 2020 IEEE 91st Vehicular Technology Conference (VTC2020-Spring). IEEE, 1--5.
[69]
Yu Zhang, Guoxu Zhou, Jing Jin, Minjue Wang, Xingyu Wang, and Andrzej Cichocki. 2013. L1-regularized multiway canonical correlation analysis for SSVEP-based BCI. IEEE transactions on neural systems and rehabilitation engineering 21, 6 (2013), 887--896.
[70]
Yang Zhang, Junhan Zhou, Gierad Laput, and Chris Harrison. 2016. Skintrack: Using the body as an electrical waveguide for continuous finger tracking on the skin. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1491--1503.
[71]
Chen Zhao, Ke-Yu Chen, Md Tanvir Islam Aumi, Shwetak Patel, and Matthew S Reynolds. 2014. SideSwipe: detecting in-air gestures around mobile devices using actual GSM signal. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 527--534.
[72]
Yongpan Zou, Jiang Xiao, Jinsong Han, Kaishun Wu, Yun Li, and Lionel M Ni. 2016. Grfid: A device-free rfid-based gesture recognition system. IEEE Transactions on Mobile Computing 16, 2 (2016), 381--393.

Cited By

View all
  • (2024)PhysioLabXR: A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality ExperimentsJournal of Open Source Software10.21105/joss.058549:93(5854)Online publication date: Jan-2024
  • (2024)LoCalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314367:4(1-27)Online publication date: 12-Jan-2024
  • (2024)A Lightweight Hand-Gesture Recognition Network With Feature Fusion Prefiltering and FMCW Radar Spatial Angle EstimationIEEE Sensors Journal10.1109/JSEN.2024.343297224:17(27926-27936)Online publication date: 1-Sep-2024
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 6, Issue 2
July 2022
1551 pages
EISSN:2474-9567
DOI:10.1145/3547347
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 July 2022
Published in IMWUT Volume 6, Issue 2

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. Cursor interaction
  2. Deep Learning
  3. In-air gestures
  4. Micro-gesture sensing
  5. Millimeter wave FMCW radar
  6. Text input

Qualifiers

  • Research-article
  • Research
  • Refereed

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)250
  • Downloads (Last 6 weeks)20
Reflects downloads up to 10 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)PhysioLabXR: A Python Platform for Real-Time, Multi-modal, Brain–Computer Interfaces and Extended Reality ExperimentsJournal of Open Source Software10.21105/joss.058549:93(5854)Online publication date: Jan-2024
  • (2024)LoCalProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36314367:4(1-27)Online publication date: 12-Jan-2024
  • (2024)A Lightweight Hand-Gesture Recognition Network With Feature Fusion Prefiltering and FMCW Radar Spatial Angle EstimationIEEE Sensors Journal10.1109/JSEN.2024.343297224:17(27926-27936)Online publication date: 1-Sep-2024
  • (2024)A mmWave MIMO Radar-Based Gesture Recognition Using Fusion of Range, Velocity, and Angular InformationIEEE Sensors Journal10.1109/JSEN.2024.335539524:6(9124-9134)Online publication date: 15-Mar-2024
  • (2024)MMHTSR: In-Air Handwriting Trajectory Sensing and Reconstruction Based on mmWave RadarIEEE Internet of Things Journal10.1109/JIOT.2023.332525811:6(10069-10083)Online publication date: 15-Mar-2024
  • (2023)HeadarProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/36109007:3(1-28)Online publication date: 27-Sep-2023
  • (2023)RF-CMProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808597:1(1-28)Online publication date: 28-Mar-2023
  • (2023)mSilentProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808387:1(1-28)Online publication date: 28-Mar-2023
  • (2023)LT-FallProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35808357:1(1-24)Online publication date: 28-Mar-2023

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media