research-article

Open access

Mental Workload and Language Production in Non-Native Speaker IPA Interaction

Authors:

Justin Edwards,

Philip R. Doyle,

Benjamin R. CowanAuthors Info & Claims

CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

Article No.: 3, Pages 1 - 8

https://doi.org/10.1145/3405755.3406118

Published: 22 July 2020 Publication History

Abstract

Through smartphones and smart speakers, intelligent personal assistants (IPAs) have made speech a common interaction modality. With linguistic coverage and varying functionality levels, many speakers engage with IPAs using a non-native language. This may impact mental workload and patterns of language production used by non-native speakers. We present a mixed-design experiment, where native (L1) and non-native (L2) English speakers completed tasks with IPAs via smartphones and smart speakers. We found significantly higher mental workload for L2 speakers in IPA interactions. Contrary to our hypotheses, we found no significant differences between L1 and L2 speakers in number of turns, lexical complexity, diversity, or lexical adaptation when encountering errors. These findings are discussed in relation to language production and processing load increases for L2 speakers in IPA interaction.

References

[1]

Diane Mayasari Alamsaputra, Kathryn J. Kohnert, Benjamin Munson, and Joe Reichle. 2006. Synthesized speech intelligibility among native speakers and non-native speakers of English. Augmentative and Alternative Communication 22, 4 (2006), 258--268. https://doi.org/10.1080/00498250600718555

[2]

René Amalberti, Noëlle Carbonell, and Pierre Falzon. 1993. User representations of computer systems in human-computer speech interaction. International Journal of Man-Machine Studies 38, 4 (April 1993), 547--566. https://doi.org/10.1006/imms.1993.1026

Digital Library

[3]

Tawfiq Ammari, Jofish Kaye, Janice Y. Tsai, and Frank Bentley. 2019. Music, Search, and IoT: How People (Really) Use Voice Assistants. ACM Trans. Comput.-Hum. Interact. 26, 3, Article Article 17 (April 2019), 28 pages. https://doi.org/10.1145/3311956

Digital Library

[4]

Dale J Barr, Roger Levy, Christoph Scheepers, and Harry J Tily. 2013. Random effects structure for confirmatory hypothesis testing: Keep it maximal. Journal of memory and language 68, 3 (2013), 255--278.

[5]

Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. 2015. Fitting Linear Mixed-Effects Models Using lme4. Journal of Statistical Software 67, 1 (2015), 1--48. https://doi.org/10.18637/jss.v067.i01

[6]

Allan Bell. 1984. Language style as audience design. Language in Society 13, 02 (June 1984), 145. https://doi.org/10.1017/S004740450001037X

[7]

Linda Bell and Joakim Gustafson. 1999. Repetition and its phonetic realizations investigating a Swedish database of spontaneous computer directed speech. In Proceedings of the XIVth International Congress of Phonetic Sciences, Vol. 99. 1221--1224.

[8]

Holly P. Branigan, Martin J. Pickering, Jamie Pearson, Janet F. McLean, and Ash Brown. 2011. The role of beliefs in lexical alignment: Evidence from dialogs with humans and computers. Cognition 121, 1 (Oct. 2011), 41--57. https://doi.org/10.1016/j.cognition.2011.05.011

[9]

Leigh Clark, Philip Doyle, Diego Garaialde, Emer Gilmartin, Stephan Schlögl, Jens Edlund, Matthew Aylett, João Cabral, Cosmin Munteanu, Justin Edwards, and Benjamin R Cowan. 2019. The State of Speech in HCI: Trends, Themes and Challenges. Interacting with Computers 31, 4 (09 2019), 349--371. https://doi.org/10.1093/iwc/iwz016

[10]

Leigh Clark, Nadia Pantidi, Orla Cooney, Philip Doyle, Diego Garaialde, Justin Edwards, Brendan Spillane, Emer Gilmartin, Christine Murad, Cosmin Munteanu, and et al. 2019. What Makes a Good Conversation? Challenges in Designing Truly Conversational Agents. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI '19). Association for Computing Machinery, New York, NY, USA, Article Paper 475, 12 pages. https://doi.org/10.1145/3290605.3300705

[11]

Benjamin R Cowan, Holly P Branigan, Habiba Begum, Lucy McKenna, and Eva Szekely. 2017. They Know as Much as We Do: Knowledge Estimation and Partner Modelling of Artificial Partners. In CogSci 2017 - 39th Annual Meeting of the Cognitive Science Society.

[12]

Benjamin R. Cowan, Philip Doyle, Justin Edwards, Diego Garaialde, Ali Hayes-Brady, Holly P. Branigan, João Cabral, and Leigh Clark. 2019. What's in an Accent? The Impact of Accented Synthetic Speech on Lexical Choice in Human-Machine Dialogue. In Proceedings of the 1st International Conference on Conversational User Interfaces (Dublin, Ireland) (CUI '19). Association for Computing Machinery, New York, NY, USA, Article 23, 8 pages. https://doi.org/10.1145/3342775.3342786

Digital Library

[13]

Benjamin R Cowan, Nadia Pantidi, David Coyle, Kellie Morrissey, Peter Clarke, Sara Al-Shehri, David Earley, and Natasha Bandeira. 2017. What can i help you with?: infrequent users' experiences of intelligent personal assistants. In Proceedings of the 19th International Conference on Human-Computer Interaction with Mobile Devices and Services. ACM, New York, NY, USA, 43.

Digital Library

[14]

S Dornic. 1979. Information processing in bilinguals: Some selected issues. Psychological Research 40, 4 (1979), 329--348.

[15]

Zoltán Dörnyei and Judit Kormos. 1998. Problem-solving mechanisms in L2 communication: A psycholinguistic perspective. Studies in second language acquisition 20, 3 (1998), 349--385.

[16]

Philip R. Doyle, Justin Edwards, Odile Dumbleton, Leigh Clark, and Benjamin R. Cowan. 2019. Mapping Perceptions of Humanness in Intelligent Personal Assistant Interaction. In Proceedings of the 21st International Conference on Human-Computer Interaction with Mobile Devices and Services (Taipei, Taiwan) (MobileHCI '19). Association for Computing Machinery, New York, NY, USA, Article 5, 12 pages. https://doi.org/10.1145/3338286.3340116

[17]

Mateusz Dubiel, Martin Halvey, and Leif Azzopardi. 2018. A Survey Investigating Usage of Virtual Personal Assistants. CoRR abs/1807.04606 (2018). arXiv:1807.04606 http://arxiv.org/abs/1807.04606

[18]

Andy P Field, Jeremy Miles, and Zoë Field. 2012. Discovering statistics using R.

[19]

Emer Gilmartin, Francesca Bonin, Loredana Cerrato, Carl Vogel, and Nick Campbell. 2015. What's the game and who's got the ball? genre in spoken interaction. In 2015 AAAI Spring Symposium Series.

[20]

Emer Gilmartin, Marine Collery, Ketong Su, Yuyun Huang, Christy Elias, Benjamin R. Cowan, and Nick Campbell. 2017. Social talk: making conversation with people and machine. In Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents - ISIAA 2017. ACM Press, Glasgow, UK, 31--32. https://doi.org/10.1145/3139491.3139494

Digital Library

[21]

Tamar H Gollan and Lori-Ann R Acenas. 2004. What is a TOT? Cognate and translation effects on tip-of-the-tongue states in Spanish-English and tagalog-English bilinguals. Journal of Experimental Psychology: Learning, Memory, and Cognition 30, 1 (2004), 246.

[22]

Pierre Guiraud. 1954. Les Charactères Statistiques du Vocabulaire. Essai de méthodologie. Presses Universitaires de France, Paris, France.

[23]

Sandra G Hart. 2006. NASA-task load index (NASA-TLX); 20 years later. In Proceedings of the human factors and ergonomics society annual meeting, Vol. 50. Sage Publications Sage CA: Los Angeles, CA, 904--908.

[24]

Sandra G Hart and Lowell E Staveland. 1988. Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research. In Advances in psychology. Vol. 52. Elsevier, 139--183.

[25]

Abhinav Jain, Minali Upreti, and Preethi Jyothi. 2018. Improved Accented Speech Recognition Using Accent Embeddings and Multi-task Learning. In Interspeech. 2454--2458.

[26]

Alan Kennedy, A Wilkes, L Elder, and Wayne Murray. 1988. Dialogue with machines. Cognition 30 (1988), 37--72. https://doi.org/10.1016/0010-0277(88)90003-0

[27]

Bret Kinsella. 2019. Google Assistant Now Supports Simplified Chinese on Android Smartphones. http://bit.ly/30Yg8qN. Accessed 27th Jan 2020.

[28]

David R. Large, Leigh Clark, Gary Burnett, Kyle Harrington, Jacob Luton, Peter Thomas, and Pete Bennett. 2019. "It's Small Talk, Jim, but Not as We Know It.": Engendering Trust through Human-Agent Conversation in an Autonomous, Self-Driving Car. In Proceedings of the 1st International Conference on Conversational User Interfaces (Dublin, Ireland) (CUI'19). Association for Computing Machinery, New York, NY, USA, Article 22, 7 pages. https://doi.org/10.1145/3342775.3342789

Digital Library

[29]

David R. Large, Leigh Clark, Annie Quandt, Gary Burnett, and Lee Skrypchuk. 2017. Steering the conversation: A linguistic exploration of natural language interactions with a digital assistant during simulated driving. Applied Ergonomics 63 (Sept. 2017), 53--61. https://doi.org/10.1016/j.apergo.2017.04.003

[30]

Patrick Mair and Rand Wilcox. 2019. Robust Statistical Methods in R Using the WRS2 Package. Behavior Research Methods (2019). Forthcoming.

[31]

Lotte Meteyard and Robert A.I. Davies. 2020. Best practice guidance for linear mixed-effects models in psychological science. Journal of Memory and Language 112 (June 2020), 104092. https://doi.org/10.1016/j.jml.2020.104092

[32]

Roger K Moore. 2017. Is spoken language all-or-nothing? Implications for future speech-based human-machine interaction. In Dialogues with Social Robots. Springer, 281--291.

[33]

Souheila Moussalli and Walcir Cardoso. 2019. Intelligent personal assistants: can they understand and be understood by accented L2 learners? Computer Assisted Language Learning 0, 0 (2019), 1--26. https://doi.org/10.1080/09588221.2019.1595664

[34]

Christie Olson and Kelli Kemery. 2019. 2019 Voice report: Consumer adoption of voice technology and digital assistants. Technical Report. Microsoft.

[35]

Lourdes Ortega. 2003. Syntactic complexity measures and their relationship to L2 proficiency: A research synthesis of college-level L2 writing. Applied linguistics 24, 4 (2003), 492--518.

[36]

Sharon Oviatt, Jon Bernard, and Gina-Anne Levow. 1998. Linguistic Adaptations During Spoken and Multimodal Error Resolution. Language and Speech 41, 3-4 (July 1998), 419--442. https://doi.org/10.1177/002383099804100409

[37]

Andrew Pawley and Frances Hodgetts Syder. 1983. Natural selection in syntax: Notes on adaptive variation and change in vernacular and literary grammar. Journal of Pragmatics 7, 5 (1983), 551--579.

[38]

Martin Porcheron, Joel E Fischer, Stuart Reeves, and Sarah Sharples. 2018. Voice Interfaces in Everyday Life. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. ACM, 640.

Digital Library

[39]

Aung Pyae and Paul Scifleet. 2018. Investigating Differences between Native English and Non-Native English Speakers in Interacting with a Voice User Interface: A Case of Google Home. In Proceedings of the 30th Australian Conference on Computer-Human Interaction (Melbourne, Australia) (OzCHI '18). Association for Computing Machinery, New York, NY, USA, 548--553. https://doi.org/10.1145/3292147.3292236

Digital Library

[40]

Aung Pyae and Paul Scifleet. 2019. Investigating the Role of User's English Language Proficiency in Using a Voice User Interface: A Case of Google Home Smart Speaker. In Extended Abstracts of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI EA '19). Association for Computing Machinery, New York, NY, USA, 6. https://doi.org/10.1145/3290607.3313038

Digital Library

[41]

R Core Team. 2018. R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria. https://www.R-project.org/

[42]

Mary Reynolds, ZS Bond, and Donald Fucci. 1996. Synthetic speech intelligibility: Comparison of native and non-native speakers of English. Augmentative and Alternative Communication 12, 1 (1996), 32--36.

[43]

Norman Segalowitz and Jan Hulstijn. [n.d.]. Automaticity in bilingualism and second language learning. Handbook of bilingualism: Psycholinguistic approaches ([n. d.]), 371--388.

[44]

Richard Towell, Roger Hawkins, and Nives Bazergui. 1996. The development of fluency in advanced learners of French. Applied linguistics 17, 1 (1996), 84--119.

[45]

Roeland Van Hout and Anne Vermeer. 2007. Comparing measures of lexical richness. Modelling and assessing vocabulary knowledge (2007), 93--115.

[46]

Catherine Watson, Wei Liu, and Bruce MacDonald. 2013. The effect of age and native speaker status on synthetic speech intelligibility.

[47]

Richard Wiese. 1984. Language Production in Foreign and Native Languages: Same or different? In Second Language Productions. Narr Verlag, Tübingen, Germany, 11--25.

[48]

Yunhan Wu, Daniel Rough, Anna Bleakley, Justin Edwards, Orla Cooney, Philip R. Doyle, Leigh Clark, and Benjamin R. Cowan. 2020. See what I'm saying? Comparing Intelligent Personal Assistant use for Native and Non-Native Language Speakers. (2020). Submitted.

Cited By

Bowman RCooney ONewbold JThieme AClark LDoherty GCowan B(2024)Exploring how politeness impacts the user experience of chatbots for mental health supportInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103181184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103181
Kann A(2022)Voice Assistants Have a Plurilingualism ProblemProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3544526(1-5)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3544526
Cihan HWu YPeña PEdwards JCowan B(2022)Bilingual by default: Voice Assistants and the role of code-switching in creating a bilingual user experienceProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3544511(1-4)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3544511
Show More Cited By

Index Terms

Mental Workload and Language Production in Non-Native Speaker IPA Interaction
1. Human-centered computing
  1. Human computer interaction (HCI)

Recommendations

Comparing Command Construction in Native and Non-Native Speaker IPA Interaction through Conversation Analysis
CUI '22: Proceedings of the 4th Conference on Conversational User Interfaces

Intelligent Personal Assistants (IPAs) are limited in the languages they support, meaning many people are left to interact using a non-native language. Yet, we know little about how people interact with IPAs in this way. Through a conversation analysis (...
See What I’m Saying? Comparing Intelligent Personal Assistant Use for Native and Non-Native Language Speakers
MobileHCI '20: 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services

Limited linguistic coverage for Intelligent Personal Assistants (IPAs) means that many interact in a non-native language. Yet we know little about how IPAs currently support or hinder these users. Through native (L1) and non-native (L2) English ...
Native speaker perceptions of (non-)native prominence patterns

The proficiency of Spanish learners of Dutch affects perceptions by Dutch listeners.They judge accentedness and nativeness more severely than comprehensibility, and can assess nativeness based solely on pitch accent distributions used to mark focus.They ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

CUI '20: Proceedings of the 2nd Conference on Conversational User Interfaces

July 2020

271 pages

ISBN:9781450375443

DOI:10.1145/3405755

Conference Chairs:
María Inés Torres,
Stephan Schlögl,
Program Chairs:
Leigh Clark,
Martin Porcheron

Copyright © 2020 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Conference

CUI '20

CUI '20: 2nd Conference on Conversational User Interfaces

July 22 - 24, 2020

Bilbao, Spain

Acceptance Rates

CUI '20 Paper Acceptance Rate 13 of 39 submissions, 33%;

Overall Acceptance Rate 34 of 100 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
1,281
Total Downloads

Downloads (Last 12 months)389
Downloads (Last 6 weeks)33

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bowman RCooney ONewbold JThieme AClark LDoherty GCowan B(2024)Exploring how politeness impacts the user experience of chatbots for mental health supportInternational Journal of Human-Computer Studies10.1016/j.ijhcs.2023.103181184:COnline publication date: 1-Apr-2024
https://dl.acm.org/doi/10.1016/j.ijhcs.2023.103181
Kann A(2022)Voice Assistants Have a Plurilingualism ProblemProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3544526(1-5)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3544526
Cihan HWu YPeña PEdwards JCowan B(2022)Bilingual by default: Voice Assistants and the role of code-switching in creating a bilingual user experienceProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3544511(1-4)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3544511
Wu YPorcheron MDoyle PEdwards JRough DCooney OBleakley AClark LCowan B(2022)Comparing Command Construction in Native and Non-Native Speaker IPA Interaction through Conversation AnalysisProceedings of the 4th Conference on Conversational User Interfaces10.1145/3543829.3543839(1-12)Online publication date: 26-Jul-2022
https://dl.acm.org/doi/10.1145/3543829.3543839
Bleakley ARough DRoper ALindsay SPorcheron MLee MNicholson SCowan BClark L(2022)Exploring Smart Speaker User Experience for People Who StammerProceedings of the 24th International ACM SIGACCESS Conference on Computers and Accessibility10.1145/3517428.3544823(1-10)Online publication date: 23-Oct-2022
https://dl.acm.org/doi/10.1145/3517428.3544823
Yang CLai SChen H(2022)The impact of intelligent personal assistants on learners’ autonomous learning of second language listening and speakingInteractive Learning Environments10.1080/10494820.2022.2141266(1-21)Online publication date: 7-Nov-2022
https://doi.org/10.1080/10494820.2022.2141266
da Silva TFurtado VFurtado EMendes MAlmeida VSales L(2022)How Do Illiterate People Interact with an Intelligent Voice Assistant?International Journal of Human–Computer Interaction10.1080/10447318.2022.212121940:3(584-602)Online publication date: 22-Sep-2022
https://doi.org/10.1080/10447318.2022.2121219
Dutsinma FPal DFunilkul SChan J(2022)A Systematic Review of Voice Assistant Usability: An ISO 9241–11 ApproachSN Computer Science10.1007/s42979-022-01172-33:4Online publication date: 3-May-2022
https://doi.org/10.1007/s42979-022-01172-3
Kontogiorgos DGustafson J(2021)Measuring Collaboration Load With Pupillary Responses - Implications for the Design of Instructions in Task-Oriented HRIFrontiers in Psychology10.3389/fpsyg.2021.62365712Online publication date: 20-Jul-2021
https://doi.org/10.3389/fpsyg.2021.623657
Aylett MClark LCowan BTorre I(2021)Building and Designing Expressive Speech SynthesisThe Handbook on Socially Interactive Agents10.1145/3477322.3477329(173-212)Online publication date: 10-Sep-2021
https://dl.acm.org/doi/10.1145/3477322.3477329
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents