research-article

Improved Touch-screen Inputting Using Sequence-level Prediction Generation

Authors:

Ping LiAuthors Info & Claims

WWW '20: Proceedings of The Web Conference 2020

Pages 3077 - 3083

https://doi.org/10.1145/3366423.3380080

Published: 20 April 2020 Publication History

Abstract

Recent years have witnessed the continuing growth of people’s dependence on touchscreen devices. As a result, input speed with the onscreen keyboard has become crucial to communication efficiency and user experience. In this work, we formally discuss the general problem of input expectation prediction with a touch-screen input method editor (IME). Taken input efficiency as the optimization target, we proposed a neural end-to-end candidates generation solution to handle automatic correction, reordering, insertion, deletion as well as completion. Evaluation metrics are also discussed base on real use scenarios. For a more thorough comparison, we also provide a statistical strategy for mapping touch coordinate sequences to text input candidates. The proposed model and baselines are evaluated on a real-world dataset. The experiment (conducted on the PaddlePaddle deep learning platform1) shows that the proposed model outperforms the baselines.

References

[1]

Ouais Alsharif, Tom Ouyang, Françoise Beaufays, Shumin Zhai, Thomas Breuel, and Johan Schalkwyk. 2015. Long short term memory neural network for keyboard gesture decoding. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2076–2080.

[2]

Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural Machine Translation by Jointly Learning to Align and Translate. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA.

[3]

Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin. 2003. A Neural Probabilistic Language Model. J. Mach. Learn. Res. 3(2003), 1137–1155.

Digital Library

[4]

Xiaojun Bi, Tom Ouyang, and Shumin Zhai. 2014. Both complete and correct?: multi-objective optimization of touchscreen keyboard. In CHI Conference on Human Factors in Computing Systems (CHI). Toronto, Canada, 2297–2306.

Digital Library

[5]

Xiaojun Bi and Shumin Zhai. 2013. Bayesian touch: a statistical criterion of target selection with finger touch. In The 26th Annual ACM Symposium on User Interface Software and Technology, (UIST). St. Andrews, UK, 51–60.

Digital Library

[6]

Xiaojun Bi and Shumin Zhai. 2013. Touchscreen text input. US Patent 8,405,630.

[7]

Christopher M Bishop. 2006. Pattern recognition and machine learning. springer.

[8]

Eric Brill and Robert C. Moore. 2000. An Improved Error Model for Noisy Channel Spelling Correction. In Proceedings of the 38th Annual Meeting of the Association for Computational Linguistics (ACL). Hong Kong, China.

Digital Library

[9]

Kyunghyun Cho, Bart van Merrienboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014. Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP). 1724–1734.

[10]

Junyoung Chung, Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. 2014. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv:1412.3555

[11]

Pieter-Tjerk de Boer, Dirk P. Kroese, Shie Mannor, and Reuven Y. Rubinstein. 2005. A Tutorial on the Cross-Entropy Method. Annals OR 134, 1 (2005), 19–67.

[12]

Noura Farra, Nadi Tomeh, Alla Rozovskaya, and Nizar Habash. 2014. Generalized Character-Level Spelling Error Correction. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL). Baltimore, MD, 161–167.

[13]

Andrew Fowler, Kurt Partridge, Ciprian Chelba, Xiaojun Bi, Tom Ouyang, and Shumin Zhai. 2015. Effects of Language Modeling and its Personalization on Touchscreen Typing Performance. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI). Seoul, Korea, 649–658.

Digital Library

[14]

Xavier Glorot, Antoine Bordes, and Yoshua Bengio. 2011. Deep Sparse Rectifier Neural Networks. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics (AISTATS). Fort Lauderdale, FL, 315–323.

[15]

Anirudh Goyal, Alex Lamb, Ying Zhang, Saizheng Zhang, Aaron C. Courville, and Yoshua Bengio. 2016. Professor Forcing: A New Algorithm for Training Recurrent Networks. In Advances in Neural Information Processing Systems (NIPS). Barcelona, Spain, 4601–4609.

[16]

Alex Graves. 2012. Sequence transduction with recurrent neural networks. Technical Report. arXiv:1211.3711

[17]

Kalervo Järvelin and Jaana Kekäläinen. 2000. IR evaluation methods for retrieving highly relevant documents. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). Athens, Greece, 41–48.

Digital Library

[18]

Diederik P. Kingma and Jimmy Ba. 2015. Adam: A Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA.

[19]

Kenneth Kocienda, Greg Christie, Bas Ording, Scott Forstall, Richard Williamson, and Jerome René Bellegarda. 2012. Method, device, and graphical user interface providing word recommendations for text input. US Patent 8,232,973.

[20]

Ping Li. 2009. ABC-Boost: Adaptive Base Class Boost for Multi-Class Classification. In Proceedings of the 26th Annual International Conference on Machine Learning (ICML). Montreal, Canada, 625–632.

Digital Library

[21]

Ping Li. 2010. Robust LogitBoost and Adaptive Base Class (ABC) LogitBoost. In Proceedings of the Twenty-Sixth Conference Annual Conference on Uncertainty in Artificial Intelligence (UAI). Catalina Island, CA, 302–311.

Digital Library

[22]

Ping Li, Christopher J. C. Burges, and Qiang Wu. 2007. McRank: Learning to Rank Using Multiple Classification and Gradient Boosting. In Advances in Neural Information Processing Systems (NIPS). Vancouver, Canada, 897–904.

[23]

Christian Liensberger. 2015. Context sensitive auto-correction. US Patent 9,218,333.

[24]

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, and Alan L. Yuille. 2015. Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN). In Proceedings of the 3rd International Conference on Learning Representations (ICLR). San Diego, CA.

[25]

Franz Josef Och and Hermann Ney. 2003. A Systematic Comparison of Various Statistical Alignment Models. Computational Linguistics 29, 1 (2003), 19–51.

Digital Library

[26]

Katie A. Siek, Yvonne Rogers, and Kay H. Connelly. 2005. Fat Finger Worries: How Older and Younger Users Physically Interact with PDAs. In Proceedings of the IFIP TC13 International Conference on Human-Computer Interaction (INTERACT). 267–280.

Digital Library

[27]

Oriol Vinyals, Lukasz Kaiser, Terry Koo, Slav Petrov, Ilya Sutskever, and Geoffrey E. Hinton. 2015. Grammar as a Foreign Language. In Advances in Neural Information Processing Systems (NIPS). Montreal, Canada, 2773–2781.

[28]

Tao Wang, David J. Wu, Adam Coates, and Andrew Y. Ng. 2012. End-to-end text recognition with convolutional neural networks. In Proceedings of the 21st International Conference on Pattern Recognition (ICPR). Tsukuba, Japan, 3304–3308.

[29]

Xiang Xiao, Teng Han, and Jingtao Wang. 2013. LensGesture: augmenting mobile interactions with back-of-device finger gestures. In Proceedings of the 15th ACM on International conference on multimodal interaction. ACM, 287–294.

Digital Library

[30]

Ying Yin, Tom Ouyang, Kurt Partridge, and Shumin Zhai. 2013. Making touchscreen keyboards adaptive to keys, hand postures, and individuals: a hierarchical spatial backoff model approach. In 2013 ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). Paris, France, 2775–2784.

Digital Library

[31]

Quanzeng You, Hailin Jin, Zhaowen Wang, Chen Fang, and Jiebo Luo. 2016. Image Captioning with Semantic Attention. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, 4651–4659.

[32]

Seunghak Yu, Nilesh Kulkarni, Haejun Lee, and Jihie Kim. 2018. On-Device Neural Language Model Based Word Prediction. In The 27th International Conference on Computational Linguistics (COLING): System Demonstrations. Santa Fe, NM, 128–131.

[33]

Jingyuan Zhang, Xin Wang, Yue Feng, Mingming Sun, and Ping Li. 2018. FastInput: Improving Input Efficiency on Mobile Devices. In Proceedings of the 27th ACM International Conference on Information and Knowledge Management (CIKM). Torino, Italy, 2057–2065.

Digital Library

Cited By

Fang ZRoy KMa QUzzaman ALim J(2021)Application of deep learning method in web crippling strength prediction of cold-formed stainless steel channel sections under end-two-flange loadingStructures10.1016/j.istruc.2021.05.09733(2903-2942)Online publication date: Oct-2021
https://doi.org/10.1016/j.istruc.2021.05.097
Fang ZRoy KMares JSham CChen BLim J(2021)Deep learning-based axial capacity prediction for cold-formed steel channel sections using Deep Belief NetworkStructures10.1016/j.istruc.2021.05.09633(2792-2802)Online publication date: Oct-2021
https://doi.org/10.1016/j.istruc.2021.05.096
Chen XLi XLi PWing JMadigan D(2020)Toward Communication Efficient Adaptive Gradient MethodProceedings of the 2020 ACM-IMS on Foundations of Data Science Conference10.1145/3412815.3416891(119-128)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3412815.3416891
Show More Cited By

Index Terms

Improved Touch-screen Inputting Using Sequence-level Prediction Generation
1. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Guided touch screen: enhanced eyes-free interaction
PerDis '16: Proceedings of the 5th ACM International Symposium on Pervasive Displays

Touch screen based interfaces are pervading into areas such as car and industrial machinery control systems, where more tangible, physical interfaces have historically been used. This often results in a reduction in the possibility to operate the ...
In-Car Touch Screen Interaction: Comparing Standard, Finger-Specific and Multi-Finger Interaction
PerDis '15: Proceedings of the 4th International Symposium on Pervasive Displays

In this paper, we explore a novel interaction technique for the automotive domain, distinguishing between different fingers when interacting with a touch screen, and compare it against standard and multi-finger gesture interaction. We conducted a pilot ...
Enhancing KLM (keystroke-level model) to fit touch screen mobile devices
MobileHCI '14: Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services

This PVA-shortpaper introduces an enhancement to the Keystroke-Level Model (KLM) by extending it with three new operators to describe interactions on mobile touchscreen devices. Based on Fitts's Law we modelled a performance measure estimate equation ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '20: Proceedings of The Web Conference 2020

April 2020

3143 pages

ISBN:9781450370233

DOI:10.1145/3366423

Editors:
Yennun Huang
Acadmica sinica, Taiwan
,
Irwin King
The Chinese University of Hong Kong, Hong Kong
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Maarten van Steen
University of Twente, Netherlands

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 April 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article
Research
Refereed limited

Conference

WWW '20

Sponsor:

SIGWEB

WWW '20: The Web Conference 2020

April 20 - 24, 2020

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
183
Total Downloads

Downloads (Last 12 months)25
Downloads (Last 6 weeks)4

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Fang ZRoy KMa QUzzaman ALim J(2021)Application of deep learning method in web crippling strength prediction of cold-formed stainless steel channel sections under end-two-flange loadingStructures10.1016/j.istruc.2021.05.09733(2903-2942)Online publication date: Oct-2021
https://doi.org/10.1016/j.istruc.2021.05.097
Fang ZRoy KMares JSham CChen BLim J(2021)Deep learning-based axial capacity prediction for cold-formed steel channel sections using Deep Belief NetworkStructures10.1016/j.istruc.2021.05.09633(2792-2802)Online publication date: Oct-2021
https://doi.org/10.1016/j.istruc.2021.05.096
Chen XLi XLi PWing JMadigan D(2020)Toward Communication Efficient Adaptive Gradient MethodProceedings of the 2020 ACM-IMS on Foundations of Data Science Conference10.1145/3412815.3416891(119-128)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3412815.3416891
Fan CLi PWing JMadigan D(2020)Classification Acceleration via Merging Decision TreesProceedings of the 2020 ACM-IMS on Foundations of Data Science Conference10.1145/3412815.3416886(13-22)Online publication date: 19-Oct-2020
https://dl.acm.org/doi/10.1145/3412815.3416886

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents