Simulating Humans at Scale to Evaluate Voice Interfaces for TVs: the Round-Trip System at Comcast
Authors: 
Breck Baldwin, Lauren Reese, LIming Zhang, Jan Neumann, Taylor Cassidy, Michael Pereira, G. Craig Murray, Kishorekumar Sundararajan, Yidnekachew Endale, Pramod Kadagattor, Paul Wolfe, Brian Aiken, Tony Braskich, Donte Jiggetts, Adam Sloan, Esther Vaturi, Crystal Pender, Ferhan TureAuthors Info & Claims
Breck Baldwin, Lauren Reese, LIming Zhang, Jan Neumann, Taylor Cassidy, Michael Pereira, G. Craig Murray, Kishorekumar Sundararajan, Yidnekachew Endale, Pramod Kadagattor, Paul Wolfe, Brian Aiken, Tony Braskich, Donte Jiggetts, Adam Sloan, Esther Vaturi, Crystal Pender, Ferhan TureAuthors Info & ClaimsPages 1279 - 1280
Abstract
Evaluating large-scale customer-facing voice interfaces involves a variety of challenges, such as data privacy, fairness or unintended bias, and the cost of human labor. Comcast's Xfinity Voice Remote is one such voice interface aimed at users looking to discover content on their TVs. The artificial intelligence (AI) behind the voice remote currently powers multiple voice interfaces, serving tens of millions of requests every day, from users across the globe.In this talk, we introduce a novel Round-Trip system we have built to evaluate the AI serving these voice interfaces in a semi-automated manner, providing a robust and cheap alternative to traditional quality assurance methods. We discuss five specific challenges we have encountered in Round-Trip and describe our solutions in detail.
Supplementary Material
Evaluating large-scale customer-facing voice interfaces involves a variety of challenges, such as data privacy, fairness or unintended bias, and the cost of human labor. Comcast's Xfinity Voice Remote is one such voice interface aimed at users looking to discover content on their TVs. The artificial intelligence (AI) behind the voice remote currently powers multiple voice interfaces, serving tens of millions of requests every day, from users across the globe. In this talk, we introduce a novel Round-Trip system we have built to evaluate the AI serving these voice interfaces in a semi-automated manner, providing a robust and cheap alternative to traditional quality assurance methods.We discuss five specific challenges we have encountered in Round-Trip and describe our solutions in detail.
- Download
- 1337.97 MB
Simulating Humans at Scale to Evaluate Voice Interfaces for TVs: the Round-Trip System at Comcast
- Download
- 1924.34 MB
Index Terms
- Simulating Humans at Scale to Evaluate Voice Interfaces for TVs: the Round-Trip System at Comcast
Recommendations
Voice-based Conversational UX Studies and Design
CHI EA '18: Extended Abstracts of the 2018 CHI Conference on Human Factors in Computing SystemsVoice User Interfaces are becoming ubiquitously available, providing unprecedented opportunities to advance our understanding of voice interaction in a burgeoning array of practices and settings. We invite participants to contribute work-in-progress in ...
Design and Evaluation of Voice User Interfaces: What Should One Consider?
Design, Operation and Evaluation of Mobile CommunicationsAbstractVoice user interfaces (VUI) come in various forms of software or hardware, are controlled by voice, and can help the user in their daily life. Despite VUIs being readily available on smartphones, they have a low adoption rate. This can be ...
Towards a human-computer interaction model for voice user interfaces
CLIHC '19: Proceedings of the IX Latin American Conference on Human Computer InteractionThe user interaction with computer systems has evolved over the years, from Command Line Interfaces (CLIs), Graphics User Interfaces (GUIs), Natural User Interfaces (NUIs) and actually Voice User Interfaces (VUIs). The use of VUIs is increasingly common,...
Comments
Information & Contributors
Information
Published In
February 2023
1345 pages
ISBN:9781450394079
DOI:10.1145/3539597
- General Chairs:
- Tat-Seng Chua,
- Hady Lauw,
- Program Chairs:
- Luo Si,
- Evimaria Terzi,
- Panayiotis Tsaparas
Copyright © 2023 Owner/Author.
Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 27 February 2023
Check for updates
Author Tags
Qualifiers
- Abstract
Conference
WSDM '23: The Sixteenth ACM International Conference on Web Search and Data Mining
February 27 - March 3, 2023
Singapore, Singapore
Acceptance Rates
Overall Acceptance Rate 498 of 2,863 submissions, 17%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 71Total Downloads
- Downloads (Last 12 months)16
- Downloads (Last 6 weeks)0
Reflects downloads up to 25 Dec 2024
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in