research-article

Public Access

Investigating Users' Preferences and Expectations for Always-Listening Voice Assistants

Authors:

Madiha Tabassum,

Tomasz Kosiński,

Primal Wijesekera,

Serge Egelman, and

Heather Richter LipfordAuthors Info & Claims

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, Volume 3, Issue 4

Article No.: 153, Pages 1 - 23

https://doi.org/10.1145/3369807

Published: 14 September 2020 Publication History

Abstract

Many consumers now rely on different forms of voice assistants, both stand-alone devices and those built into smartphones. Currently, these systems react to specific wake-words, such as "Alexa," "Siri," or "Ok Google." However, with advancements in natural language processing, the next generation of voice assistants could instead always listen to the acoustic environment and proactively provide services and recommendations based on conversations without being explicitly invoked. We refer to such devices as "always listening voice assistants" and explore expectations around their potential use. In this paper, we report on a 178-participant survey investigating the potential services people anticipate from such a device and how they feel about sharing their data for these purposes. Our findings reveal that participants can anticipate a wide range of services pertaining to a conversation; however, most of the services are very similar to those that existing voice assistants currently provide with explicit commands. Participants are more likely to consent to share a conversation when they do not find it sensitive, they are comfortable with the service and find it beneficial, and when they already own a stand-alone voice assistant. Based on our findings we discuss the privacy challenges in designing an always-listening voice assistant.

References

[1]

2019. "I don't own the data": End User Perceptions of Smart Home Device Data Practices and Risks. In Fifteenth Symposium on Usable Privacy and Security (SOUPS 2019). USENIX Association, Santa Clara, CA. https://www.usenix.org/conference/soups2019/presentation/tabassum

[2]

Alexa Feature Help. [n.d.]. How Does Drop In Work? https://www.amazon.com/gp/help/customer/display.html?nodeId=202153130 (last accessed on August 8, 2019).

[3]

Salvatore Andolina, Valeria Orso, Hendrik Schneider, Khalil Klouche, Tuukka Ruotsalo, Luciano Gamberini, and Giulio Jacucci. 2018. Investigating Proactive Search Support in Conversations. In Proceedings of the 2018 Designing Interactive Systems Conference (DIS '18). ACM, New York, NY, USA, 1295--1307.

Digital Library

[4]

Barry Brown, Moira Mcgregor, and Donald McMillan. 2015. Searchable Objects. 508--517.

[5]

Alexandra Canavan and George Zipperlen. 1996. CALLFRIEND American English-Non-Southern Dialect LDC96S46. Web Download. Philadelphia: Linguistic Data Consortium. https://ca.talkbank.org/access/CallFriend/eng-n.html. Accessed: 2019-03-19.

[6]

Alexandra Canavan and George Zipperlen. 1996. CALLFRIEND American English-Southern Dialect LDC96S47. Web Download. Philadelphia: Linguistic Data Consortium. https://ca.talkbank.org/access/CallFriend/eng-s.html. Accessed: 2019-03-19.

[7]

David Graff Canavan, Alexandra and George Zipperlen. 1997. CALLHOME American English Speech LDC97S42. Web Download. Philadelphia: Linguistic Data Consortium. https://ca.talkbank.org/access/CallHome/eng.html. Accessed: 2019-03-19.

[8]

Juan Pablo Carrascal, Rodrigo De Oliveira, and Mauro Cherubini. 2015. To Call or to Recall? That's the Research Question. ACM Trans. Comput.-Hum. Interact. 22, 1, Article 4 (March 2015), 30 pages. https://doi.org/10.1145/2656211

Digital Library

[9]

James Clawson, Jessica A. Pater, Andrew D. Miller, Elizabeth D. Mynatt, and Lena Mamykina. 2015. No Longer Wearing: Investigating the Abandonment of Personal Health-tracking Technologies on Craigslist. In Proceedings of the 2015 ACM International Joint Conference on Pervasive and Ubiquitous Computing (UbiComp '15). ACM, New York, NY, USA, 647--658.

Digital Library

[10]

Benjamin R. Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. "What Can I Help You with?": Infrequent Users' Experiences of Intelligent Personal Assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services (MobileHCI '17). ACM, New York, NY, USA, Article 43, 12 pages.

Digital Library

[11]

Wallace L. Chafe Charles Meyer Sandra A. Thompson Robert Englebretson Du Bois, John W. and Nii Martey. 2000-2005. Santa Barbara corpus of spoken American English, Parts 1-4. Philadelphia: Linguistic Data Consortium. http://www.linguistics.ucsb.edu/research/santa-barbara-corpus. Accessed: 2019-03-19.

[12]

Echo Device Help. 2019. Turn on Follow-Up Mode. https://www.amazon.com/gp/help/customer/display.html?nodeId=202201630.

[13]

Google Home Help. 2019. Have a conversation with Google Home. https://support.google.com/googlehome/answer/7685981?hl=en.

[14]

Jefferson Graham. 2017. 'Alexa, call 911' won't work. Here's what will. USA Today (July 2017). https://www.usatoday.com/story/tech/talkingtech/2017/07/19/alexa-cant-dial-911-but-google-alexa-and-siri-can-get-you-help/486075001/

[15]

Jonathan Kilgour, Jean Carletta, and Steve Renals. 2010. The Ambient Spotlight: Queryless Desktop Search from Meeting Speech. In Proceedings of the 2010 International Workshop on Searching Spontaneous Conversational Speech (SSCS '10). ACM, New York, NY, USA, 49--52.

Digital Library

[16]

Bastian Könings, Florian Schaub, Michael Weber, and Frank Kargl. 2010. Towards territorial privacy in smart environments. In 2010 AAAI Spring Symposium Series.

[17]

Lawrence L. Kupper and Kerry B. Hafner. 1989. On Assessing Interrater Agreement for Multiple Attribute Responses. Biometrics 45, 3 (Sept. 1989), 957. https://doi.org/10.2307/2531695

[18]

Josephine Lau, Benjamin Zimmerman, and Florian Schaub. 2018. Alexa, Are You Listening?: Privacy Perceptions, Concerns and Privacy-seeking Behaviors with Smart Speakers. Proc. ACM Hum.-Comput. Interact. 2, CSCW, Article 102 (Nov. 2018), 31 pages.

Digital Library

[19]

Hosub Lee and Alfred Kobsa. 2017. Privacy preference modeling and prediction in a simulated campuswide IoT environment. 2017 IEEE International Conference on Pervasive Computing and Communications (PerCom) (2017), 276--285.

[20]

Yuefeng Li, Xujuan Zhou, Peter Bruza, Yue Xu, and Raymond YK Lau. 2008. A two-stage text mining model for information filtering. In Proceedings of the 17th ACM conference on Information and knowledge management. ACM, 1023--1032.

Digital Library

[21]

Ewa Luger and Abigail Sellen. 2016. "Like Having a Really Bad PA": The Gulf Between User Expectation and Experience of Conversational Agents. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (CHI '16). ACM, New York, NY, USA, 5286--5297.

Digital Library

[22]

Nathan Malkin, Joe Deatrick, Allen Tong, Primal Wijesekera, Serge Egelman, and David Wagner. 2019. Privacy Attitudes of Smart Speaker Users. PoPETs 2019, 4 (2019), 250--271. https://petsymposium.org/2019/files/papers/issue4/popets-2019-0068.pdf

[23]

Nathan Malkin, Serge Egelman, and David Wagner. 2019. Privacy Controls for Always-Listening Devices. In Proceedings of the 2019 New Security Paradigms Workshop (NSPW 2019). ACM. https://doi.org/10.1145/3368860.3368867

Digital Library

[24]

Moira McGregor and John C. Tang. 2017. More to Meetings: Challenges in Using Speech-Based Technology to Support Meetings. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 2208--2220.

[25]

Donald McMillan. 2017. Implicit Interaction Through Machine Learning: Challenges in Design, Accountability, and Privacy. In Internet Science. Springer International Publishing, 352--358.

[26]

Donald McMillan and Antoine Loriette. 2015. Living with Listening Services: Privacy and Control in IoT. In Internet Science. Springer International Publishing, 100--109.

[27]

Donald McMillan, Antoine Loriette, and Barry Brown. 2015. Repurposing Conversation: Experiments with the Continuous Speech Stream. In Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (CHI '15). ACM, New York, NY, USA, 3953--3962.

Digital Library

[28]

Pierrick Milhorat, Stephan Schlögl, Gérard Chollet, Jerome Boudy, Anna Esposito, and Gianni Pelosi. 2014. Building the next generation of personal digital assistants. In 2014 1st International Conference on Advanced Technologies for Signal and Image Processing (ATSIP). IEEE, 458--463.

[29]

Todd F. Mozer and Pieter J. Verneulen. 2015. Background speech recognition assistant. US Patent 8,996,381.

[30]

Pardis Emami Naeini, Sruti Bhagavatula, Hana Habib, Martin Degeling, Lujo Bauer, Lorrie Faith Cranor, and Norman Sadeh. 2017. Privacy Expectations and Preferences in an IoT World. In Thirteenth Symposium on Usable Privacy and Security (SOUPS 2017). USENIX Association, Santa Clara, CA, 399--412. https://www.usenix.org/conference/soups2017/technical-sessions/presentation/naeini

Digital Library

[31]

Helen Nissenbaum. 2004. Privacy as contextual integrity. Wash. L. Rev. 79 (2004), 119.

[32]

Christi Olson and Kelli Kemery. 2019. Voice report: From answers to action: customer adoption of voice technology and digital assistants. https://advertiseonbing-blob.azureedge.net/blob/bingads/media/insight/whitepapers/2019/04%20apr/voice-report/bingads_2019_voicereport.pdf (last accessed on July 27, 2019).

[33]

Antti Oulasvirta, Aurora Pihlajamaa, Jukka Perkiö, Debarshi Ray, Taneli Vähäkangas, Tero Hasu, Niklas Vainio, and Petri Myllymäki. 2012. Long-term Effects of Ubiquitous Surveillance in the Home. In Proceedings of the 2012 ACM Conference on Ubiquitous Computing (UbiComp '12). ACM, New York, NY, USA, 41--50.

Digital Library

[34]

Sarah Perez. 2019. Smart speakers hit critical mass in 2018. https://techcrunch.com/2018/12/28/smart-speakers-hit-critical-mass-in-2018/ (last accessed on August 5, 2019).

[35]

Martin Porcheron, Joel E. Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 640, 12 pages.

Digital Library

[36]

Martin Porcheron, Joel E. Fischer, and Sarah Sharples. 2017. "Do Animals Have Accents?": Talking with Agents in Multi-Party Conversation. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 207--219.

Digital Library

[37]

Matthew Purver, Thomas L. Griffiths, Konrad P. Körding, and Joshua B. Tenenbaum. 2006. Unsupervised Topic Modelling for Multi-party Spoken Discourse. In Proceedings of the 21st International Conference on Computational Linguistics and the 44th Annual Meeting of the Association for Computational Linguistics (ACL-44). Association for Computational Linguistics, Stroudsburg, PA, USA, 17--24.

[38]

Yang Shi, Yang Wang, Ye Qi, John Chen, Xiaoyao Xu, and Kwan-Liu Ma. 2017. IdeaWall: Improving Creative Collaboration Through Combinatorial Visual Stimuli. In Proceedings of the 2017 ACM Conference on Computer Supported Cooperative Work and Social Computing (CSCW '17). ACM, New York, NY, USA, 594--603.

Digital Library

[39]

John M. Simpson. 2017. Home Assistant Adopter Beware: Google, Amazon Digital Assistant Patents Reveal Plans for Mass Snooping. https://www.consumerwatchdog.org/privacy-technology/home-assistant-adopter-beware-google-amazon-digital-assistant-patents-reveal. Accessed: 2019-04-04.

[40]

Stefan Sommer, Andreas Schieber, Kai Heinrich, and Andreas Hilbert. 2012. What is the Conversation About?: A Topic-Model-Based Approach for Analyzing Customer Sentiments in Twitter. International Journal of Intelligent Information Technologies 8 (01 2012), 10--25.

[41]

Mark Sullivan. 2019. Supply-chain sources say Facebook Portal sales are 'very low'. https://www.fastcompany.com/90419009/supply-chain-sources-say-facebook-portal-sales-are-very-low.

[42]

Alexandra Vtyurina and Adam Fourney. 2018. Exploring the Role of Conversational Cues in Guided Task Support with Virtual Assistants. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (CHI '18). ACM, New York, NY, USA, Article 208, 7 pages.

Digital Library

[43]

Primal Wijesekera, Arjun Baokar, Lynn Tsai, Joel Reardon, Serge Egelman, David Wagner, and Konstantin Beznosov. 2017. The feasibility of dynamically granted permissions: Aligning mobile privacy with user preferences. In 2017 IEEE Symposium on Security and Privacy (SP). IEEE, 1077--1093.

[44]

Eric Zeng, Shrirang Mare, and Franziska Roesner. 2017. End User Security & Privacy Concerns with Smart Homes. In Proceedings of the Thirteenth USENIX Conference on Usable Privacy and Security (SOUPS'17). USENIX Association, Berkeley, CA, USA, 65--80.

Digital Library

Cited By

Shalawadi SGetschmann Cvan Berkel NEchtler F(2024)Manual, Hybrid, and Automatic Privacy Covers for Smart Home CamerasProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661569(3453-3470)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661569
Jeromela JConlan O(2024)Devising Scrutable User Models for Time Management AssistantsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665182(250-255)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3665182
Zulfikar WChan SMaes P(2024)Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory AugmentationProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642450(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642450
Show More Cited By

Index Terms

Investigating Users' Preferences and Expectations for Always-Listening Voice Assistants
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. HCI design and evaluation methods
      1. User studies
  2. Ubiquitous and mobile computing
    1. Ubiquitous and mobile devices
      1. Personal digital assistants

Recommendations

Use of Intelligent Voice Assistants by Older Adults with Low Technology Use
Special Issue on HCI and the Body:?Reimagining Women's Health and Regular Papers

Voice assistants embodied in smart speakers (e.g., Amazon Echo, Google Home) enable voice-based interaction that does not necessarily rely on expertise with mobile or desktop computing. Hence, these voice assistants offer new opportunities to different ...
Read More
Age-Related Differences in Preferences for Using Voice Assistants
MuC '21: Proceedings of Mensch und Computer 2021

In the past few years, voice assistants have become broadly available in different forms of presentation and devices – not only as a personal assistant within smartphones but as smart speakers, within TV sets or as part of an in-car infotainment ...
Read More
Tickling Proactivity: Exploring the Use of Humor in Proactive Voice Assistants
MUM '23: Proceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia

With rapid advances in artificial intelligence and natural language processing, voice assistants are evolving into advanced digital personal assistants capable of complex tasks. As they become more proficient at understanding people’s behaviors, ...
Read More

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies

Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies Volume 3, Issue 4

December 2019

873 pages

EISSN:2474-9567

DOI:10.1145/3375704

Issue’s Table of Contents

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 September 2020

Published in IMWUT Volume 3, Issue 4

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Security Agency
Knut och Alice Wallenbergs Stiftelse
National Science Foundation
Center for Long-Term Cybersecurity, University of California Berkeley

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

44
Total Citations
View Citations
2,406
Total Downloads

Downloads (Last 12 months)532
Downloads (Last 6 weeks)54

Other Metrics

View Author Metrics

Citations

Cited By

Shalawadi SGetschmann Cvan Berkel NEchtler F(2024)Manual, Hybrid, and Automatic Privacy Covers for Smart Home CamerasProceedings of the 2024 ACM Designing Interactive Systems Conference10.1145/3643834.3661569(3453-3470)Online publication date: 1-Jul-2024
https://dl.acm.org/doi/10.1145/3643834.3661569
Jeromela JConlan O(2024)Devising Scrutable User Models for Time Management AssistantsAdjunct Proceedings of the 32nd ACM Conference on User Modeling, Adaptation and Personalization10.1145/3631700.3665182(250-255)Online publication date: 27-Jun-2024
https://dl.acm.org/doi/10.1145/3631700.3665182
Zulfikar WChan SMaes P(2024)Memoro: Using Large Language Models to Realize a Concise Interface for Real-Time Memory AugmentationProceedings of the CHI Conference on Human Factors in Computing Systems10.1145/3613904.3642450(1-18)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3613904.3642450
Hennekeuser DVaziri DGolchinfar DStevens G(2024)What I Don’t Like about You?: A Systematic Review of Impeding Aspects for the Usage of Conversational AgentsInteracting with Computers10.1093/iwc/iwae018Online publication date: 31-May-2024
https://doi.org/10.1093/iwc/iwae018
Leschanowsky ARech SPopp BBäckström T(2024)Evaluating privacy, security, and trust perceptions in conversational AI: A systematic reviewComputers in Human Behavior10.1016/j.chb.2024.108344159(108344)Online publication date: Oct-2024
https://doi.org/10.1016/j.chb.2024.108344
Alorini ASawad AAlharbi SIjaz KPrasad MKocaballi A(2024)Understanding Privacy in Smart Speakers: A Narrative ReviewInformation Security and Privacy10.1007/978-981-97-5101-3_8(143-157)Online publication date: 15-Jul-2024
https://doi.org/10.1007/978-981-97-5101-3_8
Klein ADeutschländer JKölln KRauschenberger MEscalona M(2024)Exploring the context of use for voice user interfacesJournal of Software: Evolution and Process10.1002/smr.261836:7Online publication date: 14-Jul-2024
https://dl.acm.org/doi/10.1002/smr.2618
Kustosch LGañán CVan't Schip MVan Eeten MParkin SCalandrino JTroncoso C(2023)Measuring up to (reasonable) consumer expectationsProceedings of the 32nd USENIX Conference on Security Symposium10.5555/3620237.3620321(1487-1504)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.5555/3620237.3620321
Ryu DJeon SHong JLee M(2023)Efficient Lp Distance Computation Using Function-Hiding Inner Product Encryption for Privacy-Preserving Anomaly DetectionSensors10.3390/s2308416923:8(4169)Online publication date: 21-Apr-2023
https://doi.org/10.3390/s23084169
Zargham NReicherts LAvanesi VRogers YMalaka R(2023)Tickling Proactivity: Exploring the Use of Humor in Proactive Voice AssistantsProceedings of the 22nd International Conference on Mobile and Ubiquitous Multimedia10.1145/3626705.3627777(294-320)Online publication date: 3-Dec-2023
https://dl.acm.org/doi/10.1145/3626705.3627777
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents