Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Learning to Recognize Handwriting Input with Acoustic Features

Published: 15 June 2020 Publication History
  • Get Citation Alerts
  • Abstract

    For mobile or wearable devices with a small touchscreen, handwriting input (instead of typing on the touchscreen) is highly desirable for efficient human-computer interaction. Previous passive acoustic-based handwriting solutions mainly focus on print-style capital input, which is inconsistent with people's daily habits and thus causes inconvenience. In this paper, we propose WritingRecorder, a novel universal text entry system that enables free-style lowercase handwriting recognition. WritingRecorder leverages the built-in microphone of the smartphones to record the handwritten sound, and then designs an adaptive segmentation method to detect letter fragments in real-time from the recorded sound. Then we design a neural network named Inception-LSTM to extract the hidden and unique acoustic pattern associated with the writing trajectory of each letter and thus classify each letter. Moreover, we adopt a word selection method based on language model, so as to recognize legislate words from all possible letter combinations. We implement WritingRecorder as an APP on mobile phones and conduct the extensive experimental evaluation. The results demonstrate that WritingRecorder works in real-time and can achieve 93.2% accuracy even for new users without collecting and training on their handwriting samples, under a series of practical scenarios.

    References

    [1]
    Nikola Banovic, Ticha Sethapakdi, Yasasvi Hari, Anind K. Dey, and Jennifer Mankoff. 2019. The Limits of Expert Text Entry Speed on Mobile Keyboards with Autocorrect. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 15, 12 pages. https://doi.org/10.1145/3338286.3340126
    [2]
    M. Chen, G. AlRegib, and B. Juang. 2016. Air-Writing Recognition-Part I: Modeling and Recognition of Characters, Words, and Connecting Motions. IEEE Transactions on Human-Machine Systems 46, 3 (June 2016), 403--413.
    [3]
    Mingshi Chen, Panlong Yang, Jie Xiong, Maotian Zhang, Youngki Lee, Chaocan Xiang, and Chang Tian. 2019. Your Table Can Be an Input Panel: Acoustic-based Device-Free Interaction Recognition. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 3, 1, Article 3 (March 2019), 21 pages. https://doi.org/10.1145/3314390
    [4]
    G. E. Dahl, D. Yu, L. Deng, and A. Acero. 2012. Context-Dependent Pre-Trained Deep Neural Networks for Large-Vocabulary Speech Recognition. IEEE Transactions on Audio, Speech, and Language Processing 20, 1 (Jan 2012), 30--42.
    [5]
    T. Deselaers, D. Keysers, J. Hosang, and H. A. Rowley. 2015. GyroPen: Gyroscopes for Pen-Input With Mobile Phones. IEEE Transactions on Human-Machine Systems 45, 2 (April 2015), 263--271.
    [6]
    H. Du, P. Li, H. Zhou, W. Gong, G. Luo, and P. Yang. 2018. WordRecorder: Accurate Acoustic-based Handwriting Recognition Using Deep Learning. In IEEE INFOCOM 2018 - IEEE Conference on Computer Communications. 1448--1456.
    [7]
    Tian Hao, Guoliang Xing, and Gang Zhou. 2013. iSleep: Unobtrusive Sleep Quality Monitoring Using Smartphones. In Proceedings of the 11th ACM Conference on Embedded Networked Sensor Systems (Roma, Italy) (SenSys '13). ACM, New York, NY, USA, Article 4, 14 pages. https://doi.org/10.1145/2517351.2517359
    [8]
    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2012. ImageNet Classification with Deep Convolutional Neural Networks. In Advances in Neural Information Processing Systems 25, F. Pereira, C. J. C. Burges, L. Bottou, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 1097--1105.
    [9]
    Sugang Li, Xiaoran Fan, Yanyong Zhang, Wade Trappe, Janne Lindqvist, and Richard E. Howard. 2017. Auto++: Detecting Cars Using Embedded Microphones in Real-Time. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 1, 3, Article 70 (Sept. 2017), 20 pages. https://doi.org/10.1145/3130938
    [10]
    Wenzhe Li and Tracy Hammond. 2011. Recognizing Text Through Sound Alone. In Proceedings of the Twenty-Fifth AAAI Conference on Artificial Intelligence (San Francisco, California) (AAAI'11). AAAI Press, 1481--1486.
    [11]
    Jian Liu, Yan Wang, Gorkem Kar, Yingying Chen, Jie Yang, and Marco Gruteser. 2015. Snooping Keystrokes with Mm-level Audio Ranging on a Single Phone. In Proceedings of the 21st Annual International Conference on Mobile Computing and Networking (Paris, France) (MobiCom '15). ACM, New York, NY, USA, 142--154. https://doi.org/10.1145/2789168.2790122
    [12]
    A. Manashty, J. Light, and H. Soleimani. 2018. A Concise Temporal Data Representation Model for Prediction in Biomedical Wearable Devices. IEEE Internet of Things Journal (2018), 1--1.
    [13]
    Wenguang Mao, Jian He, and Lili Qiu. 2016. CAT: High-precision Acoustic Motion Tracking. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (New York City, New York). ACM, New York, NY, USA, 69--81.
    [14]
    P. Molchanov, X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz. 2016. Online Detection and Classification of Dynamic Hand Gestures with Recurrent 3D Convolutional Neural Networks. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 4207--4215.
    [15]
    Lindasalwa Muda, Mumtaj Begam, and I. Elamvazuthi. 2010. Voice Recognition Algorithms using Mel Frequency Cepstral Coefficient (MFCC) and Dynamic Time Warping (DTW) Techniques. CoRR abs/1003.4083 (2010). arXiv:1003.4083
    [16]
    Rajalakshmi Nandakumar, Shyamnath Gollakota, and Nathaniel Watson. 2015. Contactless Sleep Apnea Detection on Smartphones. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). ACM, New York, NY, USA, 45--57. https://doi.org/10.1145/2742647.2742674
    [17]
    Rajalakshmi Nandakumar, Vikram Iyer, Desney Tan, and Shyamnath Gollakota. 2016. FingerIO: Using Active Sonar for Fine-Grained Finger Tracking. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (San Jose, California, USA). ACM, New York, NY, USA, 1515--1525.
    [18]
    V. Pham, T. Bluche, C. Kermorvant, and J. Louradour. 2014. Dropout Improves Recurrent Neural Networks for Handwriting Recognition. In 2014 14th International Conference on Frontiers in Handwriting Recognition. 285--290.
    [19]
    C. Plapous, C. Marro, and P. Scalart. 2006. Improved Signal-to-Noise Ratio Estimation for Speech Enhancement. IEEE Transactions on Audio, Speech, and Language Processing 14, 6 (Nov 2006), 2098--2108. https://doi.org/10.1109/TASL.2006.872621
    [20]
    Arik Poznanski and Lior Wolf. 2016. CNN-N-Gram for Handwriting Word Recognition. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR).
    [21]
    K. Qian, C. Wu, F. Xiao, Y. Zheng, Y. Zhang, Z. Yang, and Y. Liu. 2018. Acousticcardiogram: Monitoring Heartbeats using Acoustic Signals on Smart Devices. In IEEE INFOCOM 2018 - IEEE Conference on Computer Communications. 1574--1582. https://doi.org/10.1109/INFOCOM.2018.8485978
    [22]
    Y. Ren, C. Wang, J. Yang, and Y. Chen. 2015. Fine-grained sleep monitoring: Hearing your breathing with smartphones. In 2015 IEEE Conference on Computer Communications (INFOCOM). 1194--1202. https://doi.org/10.1109/INFOCOM.2015.7218494
    [23]
    Wenjie Ruan, Quan Z. Sheng, Lei Yang, Tao Gu, Peipei Xu, and Longfei Shangguan. 2016. AudioGest: Enabling Fine-grained Hand Gesture Detection by Decoding Echo Signal. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp '16). ACM, New York, NY, USA, 474--485. https://doi.org/10.1145/2971648.2971736
    [24]
    T. N. Sainath, O. Vinyals, A. Senior, and H. Sak. 2015. Convolutional, Long Short-Term Memory, fully connected Deep Neural Networks. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 4580--4584.
    [25]
    Maximilian Schrapel, Max-Ludwig Stadler, and Michael Rohs. 2018. Pentelligence: Combining Pen Tip Motion and Writing Sounds for Handwritten Digit Recognition. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (Montreal QC, Canada) (CHI '18). ACM, New York, NY, USA, Article 131, 11 pages.
    [26]
    A. Seniuk and D. Blostein. 2009. Pen Acoustic Emissions for Text and Gesture Recognition. In 2009 10th International Conference on Document Analysis and Recognition. 872--876.
    [27]
    S. Shi, Q. Wang, P. Xu, and X. Chu. 2016. Benchmarking State-of-the-Art Deep Learning Software Tools. In 2016 7th International Conference on Cloud Computing and Big Data (CCBD). 99--104.
    [28]
    Ke Sun, Wei Wang, Alex X. Liu, and Haipeng Dai. 2018. Depth Aware Finger Tapping on Virtual Displays. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services (Munich, Germany) (MobiSys '18). ACM, New York, NY, USA, 283--295. https://doi.org/10.1145/3210240.3210315
    [29]
    Ke Sun, Ting Zhao, Wei Wang, and Lei Xie. 2018. VSkin: Sensing Touch Gestures on Surfaces of Mobile Devices Using Acoustic Signals. In Proceedings of the 24th Annual International Conference on Mobile Computing and Networking (New Delhi, India) (MobiCom '18). ACM, New York, NY, USA, 591--605. https://doi.org/10.1145/3241539.3241568
    [30]
    Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014. Sequence to Sequence Learning with Neural Networks. In Advances in Neural Information Processing Systems 27, Z. Ghahramani, M. Welling, C. Cortes, N. D. Lawrence, and K. Q. Weinberger (Eds.). Curran Associates, Inc., 3104--3112. http://papers.nips.cc/paper/5346-sequence-to-sequence-learning-with-neural-networks.pdf
    [31]
    C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. 2016. Rethinking the Inception Architecture for Computer Vision. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2818--2826.
    [32]
    Junjue Wang, Kaichen Zhao, Xinyu Zhang, and Chunyi Peng. 2014. Ubiquitous Keyboard for Small Mobile Devices: Harnessing Multipath Fading for Fine-grained Keystroke Localization. In Proceedings of the 12th Annual International Conference on Mobile Systems, Applications, and Services (Bretton Woods, New Hampshire, USA) (MobiSys '14). ACM, New York, NY, USA, 14--27.
    [33]
    Wei Wang, Alex X. Liu, and Ke Sun. 2016. Device-free Gesture Tracking Using Acoustic Signals. In Proceedings of the 22Nd Annual International Conference on Mobile Computing and Networking (New York City, New York). ACM, New York, NY, USA, 82--94.
    [34]
    Elliott Wen, Winston Seah, Bryan Ng, Xuefeng Liu, and Jiannong Cao. 2016. UbiTouch: Ubiquitous Smartphone Touchpads Using Built-in Proximity and Ambient Light Sensors. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany). ACM, New York, NY, USA, 286--297.
    [35]
    YiChao Wu, Fei Yin, and ChengLin Liu. 2017. Improving handwritten Chinese text recognition using neural network language models and convolutional neural network shape models. Pattern Recognition 65 (2017), 251--264.
    [36]
    Y. Xie, F. Li, Y. Wu, S. Yang, and Y. Wang. 2019. D3-Guard: Acoustic-based Drowsy Driving Detection Using Smartphones. In IEEE INFOCOM 2019 - IEEE Conference on Computer Communications. 1225--1233. https://doi.org/10.1109/INFOCOM.2019.8737470
    [37]
    Chao Xu, Parth H. Pathak, and Prasant Mohapatra. 2015. Finger-writing with Smartwatch: A Case for Finger and Hand Gesture Recognition Using Smartwatch. In Proceedings of the 16th International Workshop on Mobile Computing Systems and Applications (HotMobile '15). ACM, New York, NY, USA, 9--14.
    [38]
    X. Xu, J. Yu, Y. Chen, Y. Zhu, and M. Li. 2018. SteerTrack: Acoustic-Based Device-Free Steering Tracking Leveraging Smartphones. In 2018 15th Annual IEEE International Conference on Sensing, Communication, and Networking (SECON). 1--9. https://doi.org/10.1109/SAHCN.2018.8397115
    [39]
    X. Xu, J. Yu, Y. Chen, Y. Zhu, S. Qian, and M. Li. 2018. Leveraging Audio Signals for Early Recognition of Inattentive Driving with Smartphones. IEEE Transactions on Mobile Computing 17, 7 (July 2018), 1553--1567. https://doi.org/10.1109/TMC.2017.2772253
    [40]
    H. Yin, A. Zhou, L. Liu, N. Wang, and H. Ma. 2019. Ubiquitous Writer: Robust Text Input for Small Mobile Devices via Acoustic Sensing. IEEE Internet of Things Journal 6, 3 (June 2019), 5285--5296. https://doi.org/10.1109/JIOT.2019.2900355
    [41]
    Tuo Yu, Haiming Jin, and Klara Nahrstedt. 2016. WritingHacker: Audio Based Eavesdropping of Handwriting via Mobile Devices. In Proceedings of the 2016 ACM International Joint Conference on Pervasive and Ubiquitous Computing (Heidelberg, Germany) (UbiComp '16). ACM, New York, NY, USA, 463--473.
    [42]
    Sangki Yun, Yi-Chao Chen, and Lili Qiu. 2015. Turning a Mobile Device into a Mouse in the Air. In Proceedings of the 13th Annual International Conference on Mobile Systems, Applications, and Services (Florence, Italy) (MobiSys '15). ACM, New York, NY, USA, 15--29. https://doi.org/10.1145/2742647.2742662
    [43]
    Sangki Yun, Yi-Chao Chen, Huihuang Zheng, Lili Qiu, and Wenguang Mao. 2017. Strata: Fine-Grained Acoustic-based Device-Free Tracking. In Proceedings of the 15th Annual International Conference on Mobile Systems, Applications, and Services (Niagara Falls, New York, USA) (MobiSys '17). ACM, New York, NY, USA, 15--28.
    [44]
    Maotian Zhang, Panlong Yang, Chang Tian, Lei Shi, Shaojie Tang, and Fu Xiao. 2015. SoundWrite: Text Input on Surfaces Through Mobile Acoustic Sensing. In Proceedings of the 1st International Workshop on Experiences with the Design and Implementation of Smart Objects (Paris, France) (SmartObjects '15). ACM, New York, NY, USA, 13--17.
    [45]
    Y. Zhang, J. Wang, W. Wang, Z. Wang, and Y. Liu. 2018. Vernier: Accurate and Fast Acoustic Motion Tracking Using Mobile Devices. In IEEE INFOCOM 2018 - IEEE Conference on Computer Communications. 1709--1717. https://doi.org/10.1109/INFOCOM.2018.8486365
    [46]
    Zengbin Zhang, David Chu, Xiaomeng Chen, and Thomas Moscibroda. 2012. SwordFight: Enabling a New Class of Phone-to-phone Action Games on Commodity Phones. In Proceedings of the 10th International Conference on Mobile Systems, Applications, and Services (Low Wood Bay, Lake District, UK) (MobiSys '12). ACM, New York, NY, USA, 1--14. https://doi.org/10.1145/2307636.2307638
    [47]
    Bing Zhou, Mohammed Elbadry, Ruipeng Gao, and Fan Ye. 2017. BatTracker: High Precision Infrastructure-free Mobile Device Tracking in Indoor Environments. In Proceedings of the 15th ACM Conference on Embedded Network Sensor Systems (Delft, Netherlands) (SenSys '17). ACM, New York, NY, USA, Article 13, 14 pages.
    [48]
    Tong Zhu, Qiang Ma, Shanfeng Zhang, and Yunhao Liu. 2014. Context-free Attacks Using Keyboard Acoustic Emanations. In Proceedings of the 2014 ACM SIGSAC Conference on Computer and Communications Security (Scottsdale, Arizona, USA) (CCS '14). ACM, New York, NY, USA, 453--464. https://doi.org/10.1145/2660267.2660296

    Cited By

    View all
    • (2024)AcouWrite: Acoustic-Based Handwriting Recognition on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2024.335148423:8(8557-8568)Online publication date: Aug-2024
    • (2024)Ultra Write: A Lightweight Continuous Gesture Input System with Ultrasonic Signals on COTS Devices2024 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom59722.2024.10494485(174-183)Online publication date: 11-Mar-2024
    • (2024)MMHTSR: In-Air Handwriting Trajectory Sensing and Reconstruction Based on mmWave RadarIEEE Internet of Things Journal10.1109/JIOT.2023.332525811:6(10069-10083)Online publication date: 15-Mar-2024
    • Show More Cited By

    Index Terms

    1. Learning to Recognize Handwriting Input with Acoustic Features

      Recommendations

      Comments

      Information & Contributors

      Information

      Published In

      cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies
      Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies  Volume 4, Issue 2
      June 2020
      771 pages
      EISSN:2474-9567
      DOI:10.1145/3406789
      Issue’s Table of Contents
      Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

      Publisher

      Association for Computing Machinery

      New York, NY, United States

      Publication History

      Published: 15 June 2020
      Published in IMWUT Volume 4, Issue 2

      Permissions

      Request permissions for this article.

      Check for updates

      Author Tags

      1. Acoustic sensing
      2. Handwriting recognition

      Qualifiers

      • Research-article
      • Research
      • Refereed

      Funding Sources

      • NSFC
      • the Funds for Creative Research Groups of China
      • Fundamental Research Funds for the Central Universities
      • National Key R&D Program of China
      • 111 Project

      Contributors

      Other Metrics

      Bibliometrics & Citations

      Bibliometrics

      Article Metrics

      • Downloads (Last 12 months)91
      • Downloads (Last 6 weeks)8

      Other Metrics

      Citations

      Cited By

      View all
      • (2024)AcouWrite: Acoustic-Based Handwriting Recognition on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2024.335148423:8(8557-8568)Online publication date: Aug-2024
      • (2024)Ultra Write: A Lightweight Continuous Gesture Input System with Ultrasonic Signals on COTS Devices2024 IEEE International Conference on Pervasive Computing and Communications (PerCom)10.1109/PerCom59722.2024.10494485(174-183)Online publication date: 11-Mar-2024
      • (2024)MMHTSR: In-Air Handwriting Trajectory Sensing and Reconstruction Based on mmWave RadarIEEE Internet of Things Journal10.1109/JIOT.2023.332525811:6(10069-10083)Online publication date: 15-Mar-2024
      • (2024)A Systematic Review of Human Activity Recognition Based on Mobile Devices: Overview, Progress and TrendsIEEE Communications Surveys & Tutorials10.1109/COMST.2024.335759126:2(890-929)Online publication date: Oct-2025
      • (2024)Universal Handwriting Recognition for Mobile Devices via Acoustic SensingWireless Sensor Networks10.1007/978-981-97-1010-2_18(244-256)Online publication date: 1-Mar-2024
      • (2023)TwinkleTwinkleProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35962387:2(1-30)Online publication date: 12-Jun-2023
      • (2023)Differentiable Neural Network Pruning to Enable Smart Applications on MicrocontrollersProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/35694686:4(1-19)Online publication date: 11-Jan-2023
      • (2023)PD-FMCW: Push the Limit of Device-Free Acoustic Sensing Using Phase Difference in FMCWIEEE Transactions on Mobile Computing10.1109/TMC.2022.316263122:8(4865-4880)Online publication date: 1-Aug-2023
      • (2023)Handwriting Recognition System Leveraging Vibration Signal on SmartphonesIEEE Transactions on Mobile Computing10.1109/TMC.2022.314817222:7(3940-3951)Online publication date: 1-Jul-2023
      • (2023) RFPad: Enabling Device-Free Handwriting Recognition With a Tag Square IEEE Transactions on Human-Machine Systems10.1109/THMS.2023.323660553:2(325-334)Online publication date: Apr-2023
      • Show More Cited By

      View Options

      Get Access

      Login options

      Full Access

      View options

      PDF

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      Media

      Figures

      Other

      Tables

      Share

      Share

      Share this Publication link

      Share on social media