research-article

MessageOnTap: A Suggestive Interface to Facilitate Messaging-related Tasks

Authors:

Karan Dhabalia,

Jason I. HongAuthors Info & Claims

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

Paper No.: 575, Pages 1 - 14

https://doi.org/10.1145/3290605.3300805

Published: 02 May 2019 Publication History

Abstract

Text messages are sometimes prompts that lead to information related tasks, e.g. checking one's schedule, creating reminders, or sharing content. We introduce MessageOnTap, a suggestive inter-face for smartphones that uses the text in a conversation to suggest task shortcuts that can streamline likely next actions. When activated, MessageOnTap uses word embeddings to rank relevant external apps, and parameterizes associated task shortcuts using key phrases mentioned in the conversation, such as times, persons, or events. MessageOnTap also tailors the auto-complete dictionary based on text in the conversation, to streamline any text input.We first conducted a month-long study of messaging behaviors(N=22) that informed our design. We then conducted a lab study to evaluate the effectiveness of MessageOnTap's suggestive interface, and found that participants can complete tasks 3.1x faster withMessageOnTap than their typical task flow.

Supplementary Material

MP4 File (pn5594.mp4)

Supplemental video

Download
36.33 MB

References

[1]

Fourth quarter and full year 2017 results conference call. https://s21.q4cdn.com/ 399680738/files/doc_financials/2017/Q4/Q4--17-Earnings-call-transcript.pdf, 2018.

[2]

Mobile app share of total digital time spent on selected content categories in the united states in june 2017. https://bit.ly/2BNemwO, 2018.

[3]

Average time spent per day with mobile messaging apps by adults in the united states 2015 to 2019. https://bit.ly/2BLkaXD, 2018.

[4]

Meet gboard: Search, gifs, emojis and more. right from your keyboard. https: //bit.ly/2gTvCXF, 2015.

[5]

Microsoft garage: Hub keyboard - an app for android and ios. https://bit.ly/ 2LBGrLV, 2016.

[6]

Find info about what's on your screen. https://bit.ly/2rYNgOC, 2015.

[7]

Microsoft beats google to the punch: Bing for android update does what now on tap will do. https://bit.ly/2GJFO3Z, 2018.

[8]

ios - siri - apple. https://www.apple.com/ios/siri/, 2018.

[9]

Meet bixby - a smarter way to use your phone. - samsung. https://www.samsung. com/us/explore/bixby/, 2018.

[10]

Google allo - a smart messaging app. https://allo.google.com/, 2016.

[11]

Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. What can i help you with?: infrequent users' experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services, page 43. ACM, 2017.

Digital Library

[12]

M now offers suggestions to make your messenger experience more useful, seamless and delightful. https://bit.ly/2o1M2PQ, 2017.

[13]

How to turn off facebook messenger's annoying m suggestions. https://bit.ly/ 2T95A2U, 2017.

[14]

Start plan? no thanks. how do i disable this annoying unwanted planner feature. https://bit.ly/2EQBu17, 2017.

[15]

David R Thomas. A general inductive approach for analyzing qualitative evaluation data. American journal of evaluation, 27(2):237--246, 2006.

[16]

F Maxwell Harper, Daniel Moy, and Joseph A Konstan. Facts or friends?: distinguishing informational and conversational questions in social q&a sites. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 759--768. ACM, 2009.

Digital Library

[17]

Meredith Ringel Morris, Jaime Teevan, and Katrina Panovich. What do people ask their social networks, and why?: a survey study of status message q&a behavior. In Proceedings of the SIGCHI conference on Human factors in computing systems, pages 1739--1748. ACM, 2010.

Digital Library

[18]

Karen Church and Nuria Oliver. Understanding mobile web and mobile search use in today's dynamic mobile landscape. In Proceedings of the 13th International Conference on Human Computer Interaction with Mobile Devices and Services, pages 67--76. ACM, 2011.

Digital Library

[19]

Frank R Bentley, S Tejaswi Peesapati, and Karen Church. I thought she would like to read it: Exploring sharing behaviors in the context of declining mobile web use. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 1893--1903. ACM, 2016.

Digital Library

[20]

Simon Khalaf. Apps solidify leadership six years into the mobile revolution. Flurry, http://www. flurry. com/bid/109749/Apps-Solidify-Leadership-Six-Years-intothe-Mobile-Revolution, 2014.

[21]

Amy K Karlson, Shamsi T Iqbal, Brian Meyers, Gonzalo Ramos, Kathy Lee, and John C Tang. Mobile taskflow in context: a screenshot study of smartphone usage. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pages 2009--2018. ACM, 2010.

Digital Library

[22]

Ying-Yu Chen, Frank Bentley, Christian Holz, and Cheng Xu. Sharing (and discussing) the moment: The conversations that occur around shared mobile media. In Proceedings of the 17th international conference on human-computer interaction with mobile devices and services, pages 264--273. ACM, 2015.

Digital Library

[23]

Karen Church, Denzil Ferreira, Nikola Banovic, and Kent Lyons. Understanding the challenges of mobile phone usage data. In Proceedings of the 17th International Conference on Human-Computer Interaction with Mobile Devices and Services, pages 504--514. ACM, 2015.

Digital Library

[24]

Uichin Lee, Joonwon Lee, Minsam Ko, Changhun Lee, Yuhwan Kim, Subin Yang, Koji Yatani, Gahgene Gweon, Kyong-Mee Chung, and Junehwa Song. Hooked on smartphones: an exploratory study on smartphone overuse among college students. In Proceedings of the 32nd annual ACM conference on Human factors in computing systems, pages 2327--2336. ACM, 2014.

Digital Library

[25]

Ziniu Hu, Yun Ma, Qiaozhu Mei, and Jian Tang. Roaming across the castle tunnels: an empirical study of inter-app navigation behaviors of android users. arXiv preprint arXiv:1706.08274, 2017.

[26]

Experience mini programs. https://open.wechat.com/cgi-bin/newreadtemplate? t=overseas_open/docs/mini-programs/development/experience#development_ experience, 2018.

[27]

imessage apps. https://developer.apple.com/imessage/, 2018.

[28]

Messenger platform. https://developers.facebook.com/docs/messenger-platform, 2018.

[29]

10 useful imessage apps for busy people. https://bit.ly/2QcWG2G, 2018.

[30]

Choonsung Shin, Jin-Hyuk Hong, and Anind K Dey. Understanding and prediction of mobile application usage for smart phones. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing, pages 173--182. ACM, 2012.

Digital Library

[31]

Chang Tan, Qi Liu, Enhong Chen, and Hui Xiong. Prediction for mobile application usage patterns.

[32]

Ricardo Baeza-Yates, Di Jiang, Fabrizio Silvestri, and Beverly Harrison. Predicting the next app that you are going to use. In Proceedings of the Eighth ACM International Conference on Web Search and Data Mining, pages 285--294. ACM, 2015.

Digital Library

[33]

Adam Fourney, Richard Mann, and Michael Terry. Query-feature graphs: bridging user vocabulary and system functionality. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 207--216. ACM, 2011.

Digital Library

[34]

Eytan Adar, Mira Dontcheva, and Gierad Laput. Commandspace: Modeling the relationships between tasks, descriptions and features. 2014.

Digital Library

[35]

Ethan Fast, William McGrath, Pranav Rajpurkar, and Michael S Bernstein. Augur: Mining human behaviors from fiction to power interactive systems. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, pages 237--247. ACM, 2016.

Digital Library

[36]

Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S Bernstein. Iris: A conversational agent for complex tasks. 2018.

[37]

Dae Hoon Park, Yi Fang, Mengwen Liu, and ChengXiang Zhai. Mobile app retrieval for social media users via inference of implicit intent in social media text. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management, pages 959--968. ACM, 2016.

Digital Library

[38]

Anjuli Kannan, Karol Kurach, Sujith Ravi, Tobias Kaufmann, Andrew Tomkins, Balint Miklos, Greg Corrado, László Lukács, Marina Ganea, Peter Young, et al. Smart reply: Automated response suggestion for email. In Proceedings of the ACM SIGKDD Conference on Knowledge Discovery and Data Mining, volume 36, pages 495--503, 2016.

Digital Library

[39]

Ning Ye, Ariel Fuxman, Vivek Ramavajjala, Sergey Nazarov, J Patrick McGregor, and Sujith Ravi. Photoreply: Automatically suggesting conversational responses to photos. In Proceedings of the 2018 World Wide Web Conference on World Wide Web, pages 1893--1899. International World Wide Web Conferences Steering Committee, 2018.

Digital Library

[40]

Joon-Gyum Kim, Chia-Wei Wu, Alvin Chiang, JeongGil Ko, and Sung-Ju Lee. A picture is worth a thousand words: Improving mobile messaging with real-time autonomous image suggestion. In Proceedings of the 17th International Workshop on Mobile Computing Systems and Applications, pages 51--56. ACM, 2016.

Digital Library

[41]

Christophe Van Gysel, Bhaskar Mitra, Matteo Venanzi, Roy Rosemarin, Grzegorz Kukla, Piotr Grudzien, and Nicola Cancedda. Reply with: Proactive recommendation of email attachments. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 327--336. ACM, 2017.

Digital Library

[42]

Earlence Fernandes, Oriana Riva, and Suman Nath. Appstract: on-the-fly app content semantics with better privacy. In Proceedings of the 22nd Annual International Conference on Mobile Computing and Networking, pages 361--374. ACM, 2016.

Digital Library

[43]

Ben Shneiderman and Pattie Maes. Direct manipulation vs. interface agents. interactions, 4(6):42--61, 1997.

Digital Library

[44]

Bill N Schilit, David M Hilbert, and Jonathan Trevor. Context-aware communication. IEEE Wireless Communications, 9(5):46--54, 2002.

Digital Library

[45]

Eric Horvitz. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 159--166. ACM, 1999.

Digital Library

[46]

Apple. 3D Touch. https://apple.co/2ofbwLI, 2018.

[47]

Google. App shortcuts. https://bit.ly/2BO6wTk, 2018.

[48]

Xin Rong, Adam Fourney, Robin N Brewer, Meredith Ringel Morris, and Paul N Bennett. Managing uncertainty in time expressions for virtual assistants. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 568--579. ACM, 2017.

Digital Library

[49]

Ramanathan Guha, Rob McCool, and Eric Miller. Semantic search. In Proceedings of the 12th international conference on World Wide Web, pages 700--709. ACM, 2003.

Digital Library

[50]

Hinrich Schütze. Introduction to information retrieval, volume 39.

[51]

Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. In Advances in neural information processing systems, pages 3111--3119, 2013.

Digital Library

[52]

Guoqing Zheng and Jamie Callan. Learning to reweight terms with distributed representations. In Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval, pages 575--584. ACM, 2015.

Digital Library

[53]

Jia Hui Liang, Vijay Ganesh, Pascal Poupart, and Krzysztof Czarnecki. Exponential recency weighted average branching heuristic for sat solvers. 2016.

[54]

Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Information processing & management, 24(5):513--523, 1988.

Digital Library

[55]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pretraining of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805, 2018.

[56]

Google knowledge graph search api. https://bit.ly/2fVg2ss, 2018. CHI 2019, May 4--9, 2019, Glasgow, Scotland Uk F. Chen et al.

[57]

Grégoire Mesnil, Yann Dauphin, Kaisheng Yao, Yoshua Bengio, Li Deng, Dilek Hakkani-Tur, Xiaodong He, Larry Heck, Gokhan Tur, Dong Yu, et al. Using recurrent neural networks for slot filling in spoken language understanding. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 23(3):530--539, 2015.

Digital Library

[58]

verything you need to know about google allo's privacy backlash. http://fortune. com/2016/09/22/google-allo-nope/, 2018.

[59]

Nicholas D Lane, Sourav Bhattacharya, Petko Georgiev, Claudio Forlivesi, Lei Jiao, Lorena Qendro, and Fahim Kawsar. Deepx: A software accelerator for low-power deep learning inference on mobile devices. In Proceedings of the 15th International Conference on Information Processing in Sensor Networks, page 23. IEEE Press, 2016.

Digital Library

[60]

Sujith Ravi. Projectionnet: Learning efficient on-device deep networks using neural projections. arXiv preprint arXiv:1708.00630, 2017.

[61]

Android interface definition language (aidl). https://bit.ly/2CI8S8x, 2018.

[62]

Toby Jia-Jun Li and Oriana Riva. Kite: Building conversational bots from mobile apps. In Proceedings of the 16th Annual International Conference on Mobile Systems, Applications, and Services, pages 96--109. ACM, 2018.

Digital Library

[63]

Tianlin Shi, Andrej Karpathy, Linxi Fan, Jonathan Hernandez, and Percy Liang. World of bits: An open-domain platform for web-based agents. In International Conference on Machine Learning, pages 3135--3144, 2017.

Digital Library

[64]

Evan Zheran Liu, Kelvin Guu, Panupong Pasupat, Tianlin Shi, and Percy Liang. Reinforcement learning on web interfaces using workflow-guided exploration. arXiv preprint arXiv:1802.08802, 2018.

Cited By

Xia HWang TGunturu AJiang PDuan WYao X(2023)CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and CollaborationProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606773(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606773
Lee HPark JLee U(2022)A Systematic Survey on Android API Usage for Data-driven Analytics with SmartphonesACM Computing Surveys10.1145/353081455:5(1-38)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3530814
Aoki TChujo RMatsui KChoi SHautasaari A(2022)EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech BalloonsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501920(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501920
Show More Cited By

Index Terms

MessageOnTap: A Suggestive Interface to Facilitate Messaging-related Tasks
1. Information systems
  1. Information systems applications
    1. Mobile information processing systems

Recommendations

Digital Forms for All: A Holistic Multimodal Large Language Model Agent for Health Data Entry

Digital forms help us access services and opportunities, but they are not equally accessible to everyone, such as older adults or those with sensory impairments. Large language models (LLMs) and multimodal interfaces offer a unique opportunity to ...
Perspectives on user experience for a nation-wide senior telehealth program
HCI '16: Proceedings of the 30th International BCS Human Computer Interaction Conference: Fusion!

Prior evidence shows that senior people accept technology and enjoy self-monitoring. Home telehealth support increases their confidence and sense of security. The use of communication technologies plays an important role avoiding isolation and ...
Understanding Barriers and Design Opportunities to Improve Healthcare and QOL for Older Adults through Voice Assistants
ASSETS '21: Proceedings of the 23rd International ACM SIGACCESS Conference on Computers and Accessibility

Voice-based Intelligent Virtual Assistants (IVAs) promise to improve healthcare management and Quality of Life (QOL) by introducing the paradigm of hands-free and eye-free interactions. However, there has been little understanding regarding the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI '19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems

May 2019

9077 pages

ISBN:9781450359702

DOI:10.1145/3290605

General Chairs:
Stephen Brewster
University of Glasgow, Scotland, UK
,
Geraldine Fitzpatrick
TU Wien, Austria
,
Program Chairs:
Anna Cox
University College London, UK
,
Vassilis Kostakos
University of Melbourne, Australia

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 May 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHI '19

Sponsor:

SIGCHI

CHI '19: CHI Conference on Human Factors in Computing Systems

May 4 - 9, 2019

Glasgow, Scotland Uk

Acceptance Rates

CHI '19 Paper Acceptance Rate 703 of 2,958 submissions, 24%;

Overall Acceptance Rate 6,199 of 26,314 submissions, 24%

Upcoming Conference

CHI 2025

Sponsor:
sigchi

ACM CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
548
Total Downloads

Downloads (Last 12 months)63
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Xia HWang TGunturu AJiang PDuan WYao X(2023)CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and CollaborationProceedings of the 36th Annual ACM Symposium on User Interface Software and Technology10.1145/3586183.3606773(1-16)Online publication date: 29-Oct-2023
https://dl.acm.org/doi/10.1145/3586183.3606773
Lee HPark JLee U(2022)A Systematic Survey on Android API Usage for Data-driven Analytics with SmartphonesACM Computing Surveys10.1145/353081455:5(1-38)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3530814
Aoki TChujo RMatsui KChoi SHautasaari A(2022)EmoBalloon - Conveying Emotional Arousal in Text Chats with Speech BalloonsProceedings of the 2022 CHI Conference on Human Factors in Computing Systems10.1145/3491102.3501920(1-16)Online publication date: 29-Apr-2022
https://dl.acm.org/doi/10.1145/3491102.3501920
Monge Roffarello ADe Russis L(2022)Understanding and Streamlining App Switching Experiences in Mobile InteractionInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2021.102735158:COnline publication date: 3-Jan-2022
https://dl.acm.org/doi/10.1016/j.ijhcs.2021.102735
Arsan DZaidi ASagar AKumar R(2021)App-Based Task Shortcuts for Virtual AssistantsThe 34th Annual ACM Symposium on User Interface Software and Technology10.1145/3472749.3474808(1089-1099)Online publication date: 10-Oct-2021
https://dl.acm.org/doi/10.1145/3472749.3474808
Griggio CSato AMackay WYatani KKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Mediating Intimacy with DearBoard: a Co-Customizable Keyboard for Everyday MessagingProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445757(1-16)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445757
Li TPopowski LMitchell TMyers BKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)Screen2Vec: Semantic Embedding of GUI Screens and GUI ComponentsProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445049(1-15)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445049
Devina SNadia AChen VGunawan A(2021)The Influence of UI UX Design to Number of Users Between ‘Line’ and ‘Whatsapp’2021 1st International Conference on Computer Science and Artificial Intelligence (ICCSAI)10.1109/ICCSAI53272.2021.9609775(27-31)Online publication date: 28-Oct-2021
https://doi.org/10.1109/ICCSAI53272.2021.9609775
Li TMitchell TMyers B(2021)Demonstration + Natural Language: Multimodal Interfaces for GUI-Based Interactive Task Learning AgentsArtificial Intelligence for Human Computer Interaction: A Modern Approach10.1007/978-3-030-82681-9_15(495-537)Online publication date: 5-Nov-2021
https://doi.org/10.1007/978-3-030-82681-9_15
Rahman SSiangliulue PMarcus A(2020)MixTAPEProceedings of the ACM on Human-Computer Interaction10.1145/34152404:CSCW2(1-26)Online publication date: 15-Oct-2020
https://dl.acm.org/doi/10.1145/3415240
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten