research-article

Public Access

App-Based Task Shortcuts for Virtual Assistants

Authors:

Ranjitha KumarAuthors Info & Claims

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

Pages 1089 - 1099

https://doi.org/10.1145/3472749.3474808

Published: 12 October 2021 Publication History

All formats PDF

Abstract

Virtual assistants like Google Assistant and Siri often interface with external apps when they cannot directly perform a task. Currently, developers must manually expose the capabilities of their apps to virtual assistants, using App Actions on Android or Shortcuts on iOS. This paper presents savant, a system that automatically generates task shortcuts for virtual assistants by mapping user tasks to relevant UI screens in apps. For a given natural language task (e.g., “send money to Joe”), savant leverages text and semantic information contained within UIs to identify relevant screens, and intent modeling to parse and map entities (e.g., “Joe”) to required UI inputs. Therefore, savant allows virtual assistants to interface with apps and handle new tasks without requiring any developer effort. To evaluate savant, we performed a user study to identify common tasks users perform with virtual assistants. We then demonstrate that savant can find relevant app screens for those tasks and autocomplete the UI inputs.

References

[1]

Eytan Adar, Mira Dontcheva, and Gierad Laput. 2014. CommandSpace: modeling the relationships between tasks, descriptions and features. In Proc. UIST. ACM, 167–176.

Digital Library

[2]

Steven Arzt, Siegfried Rasthofer, Christian Fritz, Eric Bodden, Alexandre Bartel, Jacques Klein, Yves Le Traon, Damien Octeau, and Patrick McDaniel. 2014. Flowdroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for android apps. ACM SIGPLAN Notices 49, 6 (2014), 259–269.

Digital Library

[3]

Tanzirul Azim, Oriana Riva, and Suman Nath. 2016. uLink: Enabling user-defined deep linking to app content. In Proc. MobiSys. ACM, 305–318.

Digital Library

[4]

Ricardo Baeza-Yates, Di Jiang, Fabrizio Silvestri, and Beverly Harrison. 2015. Predicting the next app that you are going to use. In Proc. WSDM. ACM, 285–294.

Digital Library

[5]

Matthias Böhmer, Brent Hecht, Johannes Schöning, Antonio Krüger, and Gernot Bauer. 2011. Falling asleep with Angry Birds, Facebook and Kindle: a large scale study on mobile application usage. In Proc. MobileHCI. ACM, 47–56.

Digital Library

[6]

Juan Pablo Carrascal and Karen Church. 2015. An in-situ study of mobile app & mobile search interactions. In Proc. CHI. ACM, 2739–2748.

Digital Library

[7]

Fanglin Chen, Kewei Xia, Karan Dhabalia, and Jason I Hong. 2019. MessageOnTap: A Suggestive Interface to Facilitate Messaging-related Tasks. In Proc. CHI. ACM, 575.

Digital Library

[8]

Karen Church, Denzil Ferreira, Nikola Banovic, and Kent Lyons. 2015. Understanding the challenges of mobile phone usage data. In Proc. MobileHCI. ACM, 504–514.

Digital Library

[9]

Google Cloud. 2021. Dialogflow. https://cloud.google.com/dialogflow

[10]

Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. What can i help you with?: infrequent users’ experiences of intelligent personal assistants. In Proc. MobileHCI. ACM, 43.

Digital Library

[11]

Biplab Deka, Zifeng Huang, Chad Franzen, Joshua Hibschman, Daniel Afergan, Yang Li, Jeffrey Nichols, and Ranjitha Kumar. 2017. Rico: A mobile app dataset for building data-driven design applications. In Proc. UIST. ACM, 845–854.

Digital Library

[12]

Biplab Deka, Zifeng Huang, and Ranjitha Kumar. 2016. ERICA: Interaction mining mobile apps. In Proc. UIST. ACM, 767–776.

Digital Library

[13]

Android Developers. 2021. App Actions. https://developers.google.com/actions/appactions

[14]

Android Developers. 2021. Built-in intents. https://developers.google.com/actions/reference/built-in-intents/

[15]

Android Developers. 2021. Handling Android App Links. https://developer.android.com/training/app-links

[16]

Android Developers. 2021. Intents and Intent Filters. https://developer.android.com/guide/components/intents-filters

[17]

Android Developers. 2021. Introduction to Activities. https://developer.android.com/guide/components/activities/intro-activities

[18]

Google Developers. 2021. Play Protect. https://developers.google.com/android/play-protect

[19]

Trinh Minh Tri Do, Jan Blom, and Daniel Gatica-Perez. 2011. Smartphone usage in the wild: a large-scale analysis of applications and context. In Proc. ICMI. ACM, 353–360.

[20]

Apple Developer Documentation. 2021. Donating Shortcuts. https://developer.apple.com/documentation/sirikit/donating_shortcuts

[21]

Apple Developer Documentation. 2021. Siri Shortcuts. https://developer.apple.com/design/human-interface-guidelines/sirikit/overview/siri-shortcuts/

[22]

Apple Developer Documentation. 2021. Suggesting Shortcuts to Users. https://developer.apple.com/documentation/sirikit/shortcut_management/suggesting_shortcuts_to_users

[23]

William Enck, Peter Gilbert, Seungyeop Han, Vasant Tendulkar, Byung-Gon Chun, Landon P Cox, Jaeyeon Jung, Patrick McDaniel, and Anmol N Sheth. 2014. Taintdroid: an information-flow tracking system for realtime privacy monitoring on smartphones. ACM Transactions on Computer Systems 32, 2 (2014), 1–29.

Digital Library

[24]

Ethan Fast, Binbin Chen, Julia Mendelsohn, Jonathan Bassen, and Michael S Bernstein. 2018. Iris: A conversational agent for complex tasks. In Proc. CHI. ACM, 473.

Digital Library

[25]

Adam Fourney, Richard Mann, and Michael Terry. 2011. Query-feature graphs: bridging user vocabulary and system functionality. In Proc. UIST. ACM, 207–216.

Digital Library

[26]

Alessandra Gorla, Ilaria Tavecchia, Florian Gross, and Andreas Zeller. 2014. Checking app behavior against app descriptions. In Proc. ICSE. 1025–1035.

Digital Library

[27]

Shuai Hao, Bin Liu, Suman Nath, William GJ Halfond, and Ramesh Govindan. 2014. PUMA: programmable UI-automation for large-scale dynamic analysis of mobile apps. In Proc. MobiSys. ACM, 204–217.

Digital Library

[28]

Google Assistant Help. 2019. Find info about what’s on your screen. https://support.google.com/assistant/answer/7393909

[29]

Ziniu Hu, Yun Ma, Qiaozhu Mei, and Jian Tang. 2017. Roaming across the castle tunnels: An empirical study of inter-app navigation behaviors of Android users. arXiv preprint arXiv:1706.08274(2017).

[30]

Jianjun Huang, Xiangyu Zhang, Lin Tan, Peng Wang, and Bin Liang. 2014. Asdroid: Detecting stealthy behaviors in android applications by user interface and program behavior contradiction. In Proc. ICSE. 1036–1046.

Digital Library

[31]

IFTTT. 2021. Every thing works better together - IFTTT. https://ifttt.com/

[32]

Alexandros Karatzoglou, Linas Baltrunas, Karen Church, and Matthias Böhmer. 2012. Climbing the app wall: enabling mobile app discovery through context-aware recommendations. In Proc. CIKM. ACM, 2527–2530.

Digital Library

[33]

Donghwi Kim, Sooyoung Park, Jihoon Ko, Steven Y Ko, and Sung-Ju Lee. 2019. X-Droid: A Quick and Easy Android Prototyping Framework with a Single-App Illusion. In Proc. UIST. 95–108.

Digital Library

[34]

Uichin Lee, Joonwon Lee, Minsam Ko, Changhun Lee, Yuhwan Kim, Subin Yang, Koji Yatani, Gahgene Gweon, Kyong-Mee Chung, and Junehwa Song. 2014. Hooked on smartphones: an exploratory study on smartphone overuse among college students. In Proc. CHI. ACM, 2327–2336.

Digital Library

[35]

Huoran Li, Xuan Lu, Xuanzhe Liu, Tao Xie, Kaigui Bian, Felix Xiaozhu Lin, Qiaozhu Mei, and Feng Feng. 2015. Characterizing smartphone usage patterns from millions of android users. In Proc. IMC. ACM, 459–472.

Digital Library

[36]

Toby Jia-Jun Li, Amos Azaria, and Brad A Myers. 2017. SUGILITE: creating multimodal smartphone automation by demonstration. In Proc. CHI. ACM, 6038–6049.

[37]

Toby Jia-Jun Li, Marissa Radensky, Justin Jia, Kirielle Singarajah, Tom M Mitchell, and Brad A Myers. 2019. PUMICE: A Multi-Modal Agent that Learns Concepts and Conditionals from Natural Language and Demonstrations. In Proc. UIST. 577–589.

[38]

Toby Jia-Jun Li and Oriana Riva. 2018. KITE: Building conversational bots from mobile apps. In Proc. MobiSys. ACM, 96–109.

[39]

Yuanchun Li, Ziyue Yang, Yao Guo, and Xiangqun Chen. 2019. Humanoid: A deep learning-based approach to automated black-box android app testing. In Proc. ASE. IEEE, 1070–1073.

Digital Library

[40]

Bin Liu, Deguang Kong, Lei Cen, Neil Zhenqiang Gong, Hongxia Jin, and Hui Xiong. 2015. Personalized mobile app recommendation: Reconciling app functionality and user privacy preference. In Proc. WSDM. ACM, 315–324.

Digital Library

[41]

Thomas F Liu, Mark Craft, Jason Situ, Ersin Yumer, Radomir Mech, and Ranjitha Kumar. 2018. Learning design semantics for mobile apps. In Proc. UIST. ACM, 569–579.

Digital Library

[42]

Yun Ma, Ziniu Hu, Yunxin Liu, Tao Xie, and Xuanzhe Liu. 2018. Aladdin: Automating Release of Deep-Link APIs on Android. In Proc. WWW. International World Wide Web Conferences Steering Committee, 1469–1478.

Digital Library

[43]

Yun Ma, Xuanzhe Liu, Meihua Yu, Yunxin Liu, Qiaozhu Mei, and Feng Feng. 2015. Mash droid: An approach to mobile-oriented dynamic services discovery and composition by in-app search. In Proc. ICWS. IEEE, 725–730.

Digital Library

[44]

Microsoft. 2019. 2019 Voice report: Consumer adoption of voice technology and digital assistants. https://about.ads.microsoft.com/en-us/insights/2019-voice-report

[45]

Yuhong Nan, Min Yang, Zhemin Yang, Shunfan Zhou, Guofei Gu, and XiaoFeng Wang. 2015. Uipicker: User-input privacy identification in mobile applications. In Proc. {USENIX} Security Symposium. 993–1008.

[46]

Jordan Novett. 2019. Microsoft beats Google to the punch: Bing for Android update does what Now on Tap will do. https://venturebeat.com/2015/08/20/microsoft-beats-google-to-the-punch-bing-for-android-update-does-what-now-on-tap-will-do/

[47]

European Parliament and Council of the European Union. 2018. General Data Protection Regulation. https://eur-lex.europa.eu/eli/reg/2016/679/oj

[48]

Rodrigo Pimentel. 2021. Chatito. https://github.com/rodrigopivi/Chatito

[49]

Xin Rong, Adam Fourney, Robin N Brewer, Meredith Ringel Morris, and Paul N Bennett. 2017. Managing uncertainty in time expressions for virtual assistants. In Proc. CHI. ACM, 568–579.

Digital Library

[50]

Alborz Rezazadeh Sereshkeh, Gary Leung, Krish Perumal, Caleb Phillips, Minfan Zhang, Afsaneh Fazly, and Iqbal Mohomed. 2020. VASTA: a vision and language-assisted smartphone task automation system. In Proc. IUI. 22–32.

Digital Library

[51]

Choonsung Shin, Jin-Hyuk Hong, and Anind K Dey. 2012. Understanding and prediction of mobile application usage for smart phones. In Proc. Ubicomp. ACM, 173–182.

Digital Library

[52]

Apple Support. 2021. Use Siri Shortcuts. https://support.apple.com/en-us/HT209055

[53]

Chang Tan, Qi Liu, Enhong Chen, and Hui Xiong. 2012. Prediction for mobile application usage patterns. In Nokia MDC Workshop, Vol. 12.

[54]

Workflow. 2021. Workflow - Powerful automation made simple. https://workflow.is/

[55]

Qiang Xu, Jeffrey Erman, Alexandre Gerber, Zhuoqing Mao, Jeffrey Pang, and Shobha Venkataraman. 2011. Identifying diverse usage behaviors of smartphone apps. In Proc. IMC. ACM, 329–344.

Digital Library

[56]

Bo Yan and Guanling Chen. 2011. AppJoy: personalized mobile application discovery. In Proc. MobiSys. ACM, 113–126.

Digital Library

[57]

Sha Zhao, Julian Ramos, Jianrong Tao, Ziwen Jiang, Shijian Li, Zhaohui Wu, Gang Pan, and Anind K Dey. 2016. Discovering different kinds of smartphone users through their application usage behaviors. In Proc. UbiComp. ACM, 498–509.

Digital Library

[58]

Yu Zhong, TV Raman, Casey Burkhardt, Fadi Biadsy, and Jeffrey P Bigham. 2014. JustSpeak: enabling universal voice control on Android. In Proc. W4A. 1–4.

Digital Library

[59]

Hengshu Zhu, Hui Xiong, Yong Ge, and Enhong Chen. 2014. Mobile app recommendations with security and privacy awareness. In Proc. KDD. ACM, 951–960.

Digital Library

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Huang FLi GLi TLi Y(2024)Automatic Macro Mining from Interaction Traces at ScaleProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642074(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642074
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Show More Cited By

Recommendations

An Explorative Study of the Mobile App Ecosystem from App Developers' Perspective
WWW '17: Proceedings of the 26th International Conference on World Wide Web

With the prevalence of smartphones, app markets such as Apple App Store and Google Play has become the center stage in the mobile app ecosystem, with millions of apps developed by tens of thousands of app developers in each major market. This paper ...
Inter-app communication between Android apps developed in app-inventor and Android studio
MOBILESoft '16: Proceedings of the International Conference on Mobile Software Engineering and Systems

Communications between mobile apps are an important aspect of mobile platforms. Android is specifically designed with inter-app communication in mind and depends on this to provide different platform specific functionalities. Android Apps can either be ...
How to be a successful app developer: lessons from the simulation of an app ecosystem
GECCO '12: Proceedings of the 14th annual conference on Genetic and evolutionary computation

App developers are constantly competing against each other to win more downloads for their apps. With hundreds of thousands of apps in these online stores, what strategy should a developer use to be successful? Should they innovate, make many similar ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 2021

1357 pages

ISBN:9781450386357

DOI:10.1145/3472749

Editors:
Jeffrey Nichols
Apple, USA
,
Ranjitha Kumar
UIUC, USA
,
Michael Nebeling
University of Michigan, USA

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

UIST '21

Sponsor:

UIST '21: The 34th Annual ACM Symposium on User Interface Software and Technology

October 10 - 14, 2021

Virtual Event, USA

Acceptance Rates

Overall Acceptance Rate 561 of 2,567 submissions, 22%

Upcoming Conference

UIST '25

Sponsor:
sigchi
sigchi

The 38th Annual ACM Symposium on User Interface Software and Technology

September 28 - October 1, 2025

Busan , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
1,024
Total Downloads

Downloads (Last 12 months)348
Downloads (Last 6 weeks)63

Reflects downloads up to 01 Mar 2025

Other Metrics

View Author Metrics

Citations

Cited By

Cho HYan YTodi KParent MSmith MJonker TBenko HLindlbauer D(2024)MineXR: Mining Personalized Extended Reality InterfacesProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642394(1-17)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642394
Huang FLi GLi TLi Y(2024)Automatic Macro Mining from Interaction Traces at ScaleProceedings of the 2024 CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642074(1-16)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642074
Dubiel MBarghouti YKudryavtseva KLeiva L(2024)On-device query intent prediction with lightweight LLMs to support ubiquitous conversationsScientific Reports10.1038/s41598-024-63380-614:1Online publication date: 3-Jun-2024
https://doi.org/10.1038/s41598-024-63380-6
Heo JLee U(2023)Form to Flow: Exploring Challenges and Roles of Conversational UX Designers in Real-world, Multi-channel Service EnvironmentsProceedings of the ACM on Human-Computer Interaction10.1145/36101897:CSCW2(1-24)Online publication date: 4-Oct-2023
https://dl.acm.org/doi/10.1145/3610189

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten