poster

Fake Moods: Can Users Trick an Emotion-Aware VoiceBot?

Authors:

Andreas ButzAuthors Info & Claims

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

Article No.: 269, Pages 1 - 4

https://doi.org/10.1145/3411763.3451744

Published: 08 May 2021 Publication History

Abstract

The ability to deal properly with emotion could be a critical feature of future VoiceBots. Humans might even choose to use fake emotions, e.g., sound angry to emphasize what they are saying or sound nice to get what they want. However, it is unclear whether current emotion detection methods detect such acted emotions properly, or rather the true emotion of the speaker. We asked a small number of participants (26) to mimic five basic emotions and used an open source emotion-in-voice detector to provide feedback on whether their acted emotion was recognized as intended. We found that it was difficult for participants to mimic all five emotions and that certain emotions were easier to mimic than others. However, it remains unclear whether this is due to the fact that emotion was only acted or due to the insufficiency of the detection software. As an intended side effect, we collected a small corpus of labeled data for acted emotion in speech, which we plan to extend and eventually use as training data for our own emotion detection. We present the study setup and discuss some insights on our results.

References

[1]

Christos-Nikolaos Anagnostopoulos, Theodoros Iliou, and Ioannis Giannoukos. 2015. Features and classifiers for emotion recognition from speech: a survey from 2000 to 2011. Artificial Intelligence Review 43, 2 (2015), 155–177.

Digital Library

[2]

Rachel Batish. 2018. Voicebot and Chatbot Design: Flexible Conversational Interfaces with Amazon Alexa, Google Home, and Facebook Messenger. Packt Publishing Ltd, Birmingham, UK.

[3]

Carlos Busso, Murtaza Bulut, Chi-Chun Lee, Abe Kazemzadeh, Emily Mower, Samuel Kim, Jeannette N Chang, Sungbok Lee, and Shrikanth S Narayanan. 2008. IEMOCAP: Interactive emotional dyadic motion capture database. Language resources and evaluation 42, 4 (2008), 335.

[4]

Moataz El Ayadi, Mohamed S Kamel, and Fakhri Karray. 2011. Survey on speech emotion recognition: Features, classification schemes, and databases. Pattern Recognition 44, 3 (2011), 572–587.

Digital Library

[5]

Haytham M Fayek, Margaret Lech, and Lawrence Cavedon. 2017. Evaluating deep learning architectures for Speech Emotion Recognition. Neural Networks 92(2017), 60–68.

[6]

Sanaul Haq, Philip JB Jackson, and James Edge. 2008. Audio-visual feature selection and reduction for emotion classification. In Auditory-Visual Speech Processing (AVSP’08). ISCA, Queensland, Australia, 185–190.

[7]

Sanaul Haq, Philip JB Jackson, and J Edge. 2009. Speaker-dependent audio-visual emotion recognition. In Auditory-Visual Speech Processing (AVSP’09). ISCA, Norwich, UK, 53–58.

[8]

Yanzhang He, Tara N Sainath, Rohit Prabhavalkar, Ian McGraw, Raziel Alvarez, Ding Zhao, David Rybach, Anjuli Kannan, Yonghui Wu, Ruoming Pang, 2019. Streaming end-to-end speech recognition for mobile devices. In The IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP’19). IEEE, Brighton, 6381–6385.

[9]

Brandi House, Jonathan Malkin, and Jeff Bilmes. 2009. The VoiceBot: a voice controlled robot arm. In The ACM SIGCHI Conference on Human Factors in Computing Systems. ACM, Boston, USA, 183–192.

Digital Library

[10]

Patrik N Juslin, Petri Laukka, and Tanja Bänziger. 2018. The mirror to our soul? Comparisons of spontaneous and posed vocal expression of emotion. Journal of nonverbal behavior 42, 1 (2018), 1–40.

[11]

Krishna Mohan Kudiri, Abas Md Said, and M Yunus Nayan. 2016. Human emotion detection through speech and facial expressions. In The International Conference on Computer and Information Sciences (ICCOINS’16). IEEE, Kuala Lumpur, Malaysia, 351–356.

[12]

S Lalitha and Shikha Tripathi. 2016. Emotion detection using perceptual based speech features. In The IEEE Annual India Conference (INDICON’16). IEEE, Bangalore, India, 1–5.

[13]

Adi Lausen and Kurt Hammerschmidt. 2020. Emotion recognition and confidence ratings predicted by vocal stimulus type and prosodic parameters. Humanities and Social Sciences Communications 7, 1(2020), 1–17.

[14]

Chul Min Lee, Serdar Yildirim, Murtaza Bulut, Abe Kazemzadeh, Carlos Busso, Zhigang Deng, Sungbok Lee, and Shrikanth Narayanan. 2004. Emotion recognition based on phoneme classes. In The International Conference on Spoken Language Processing(ICSLP’04). ISCA, Jeju Island, Korea.

[15]

Ruru Li, Dali Yang, Xinxing Li, Renyu Wang, Mingxing Xu, and Thomas Fang Zheng. 2016. Relative entropy normalized Gaussian supervector for speech emotion recognition using kernel extreme learning machine. In The Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA’16). IEEE, Jeju, Korea (South), 1–5.

[16]

Wootaek Lim, Daeyoung Jang, and Taejin Lee. 2016. Speech emotion recognition using convolutional and recurrent neural networks. In The Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA’16). IEEE, Jeju, Korea (South), 1–4.

[17]

Endang Wahyu Pamungkas. 2019. Emotionally-aware chatbots: A survey. arXiv preprint arXiv:1906.09774 abs/1906.09774 (2019).

[18]

Marc D Pell, Laura Monetta, Silke Paulmann, and Sonja A Kotz. 2009. Recognizing emotions in a foreign language. Journal of Nonverbal Behavior 33, 2 (2009), 107–120.

[19]

Marc D Pell, Silke Paulmann, Chinar Dara, Areej Alasseri, and Sonja A Kotz. 2009. Factors in the recognition of vocally expressed emotions: A comparison of four languages. Journal of Phonetics 37, 4 (2009), 417–435.

[20]

Isidoros Perikos and Ioannis Hatzilygeroudis. 2013. Recognizing emotion presence in natural language sentences. In The International conference on engineering applications of neural networks. Springer, Halkidiki, Greece, 30–39.

[21]

Rosalind W Picard. 2000. Affective computing. MIT press, Boston, USA.

Digital Library

[22]

S Revathy 2020. Health Care Counselling Via Voicebot Using Multinomial Naive Bayes Algorithm. In The International Conference on Communication and Electronics Systems (ICCES’20). IEEE, Coimbatore, India, 1063–1067.

[23]

Björn W Schuller. 2018. Speech emotion recognition: Two decades in a nutshell, benchmarks, and ongoing trends. Commun. ACM 61, 5 (2018), 90–99.

Digital Library

[24]

Masaaki Takebe, Kazumasa Yamamoto, and Seiichi Nakagawa. 2016. Investigation of glottal features and annotation procedures for speech emotion recognition. In The Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA’16). IEEE, Jeju, Korea (South), 1–4.

[25]

Sarah Theres Völkel, Renate Haeuslschmid, Anna Werner, Heinrich Hussmann, and Andreas Butz. 2020. How to Trick AI: Users’ Strategies for Protecting Themselves from Automatic Personality Assessment. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems (Honolulu, HI, USA) (CHI ’20). Association for Computing Machinery, New York, NY, USA, 1–15. https://doi.org/10.1145/3313831.3376877

Digital Library

[26]

Frank Wilcoxon. 1992. Individual comparisons by ranking methods. In Breakthroughs in statistics. Springer, New York,USA, 196–202.

[27]

Jianfeng Zhao, Xia Mao, and Lijiang Chen. 2019. Speech emotion recognition using deep 1D & 2D CNN LSTM networks. Biomedical Signal Processing and Control 47 (2019), 312–323.

Cited By

Anisha SSen ABain C(2024)Evaluating the Potential and Pitfalls of AI-Powered Conversational Agents as Humanlike Virtual Health Carers in the Remote Management of Noncommunicable Diseases: Scoping ReviewJournal of Medical Internet Research10.2196/5611426(e56114)Online publication date: 16-Jul-2024
https://doi.org/10.2196/56114
Ma YNordberg OZhang YRongve ABachinski MFjeld M(2024)Understanding Dementia Speech: Towards an Adaptive Voice Assistant for Enhanced CommunicationCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661326(15-21)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661326
Vogel CAhmad K(2023)Agreement and disagreement between major emotion recognition systemsKnowledge-Based Systems10.1016/j.knosys.2023.110759276:COnline publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1016/j.knosys.2023.110759
Show More Cited By

Recommendations

Emotion-Aware Voice Assistants: Design, Implementation, and Preliminary Insights
CHCHI '23: Proceedings of the Eleventh International Symposium of Chinese CHI

The increasing prevalence of voice assistants in daily life has transformed human-technology interaction. However, current voice assistants fall short in their ability to comprehend users’ emotional states through speech and respond empathetically. This ...
Changing induced moods via virtual reality
PERSUASIVE'06: Proceedings of the First international conference on Persuasive technology for human well-being

Mood Induction Procedures (MIPs) are designed to induce emotional changes in experimental subjects in a controlled way, manipulating variables inside the laboratory. The induced mood should be an experimental analogue of the mood that would occur in a ...
Toward emotion aware computing: an integrated approach using multichannel neurophysiological recordings and affective visual stimuli
Special section on new and emerging technologies in bioinformatics and bioengineering

This paper proposes a methodology for the robust classification of neurophysiological data into four emotional states collected during passive viewing of emotional evocative pictures selected from the International Affective Picture System. The proposed ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHI EA '21: Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems

May 2021

2965 pages

ISBN:9781450380959

DOI:10.1145/3411763

Editors:
Yoshifumi Kitamura
Tohoku University, Japan
,
Aaron Quigley
University of New South Wales, Australia
,
Katherine Isbister
University of California Santa Cruz, USA
,
Takeo Igarashi
The University of Tokyo, Japan

Copyright © 2021 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 May 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Poster
Research
Refereed limited

Conference

CHI '21

Sponsor:

SIGCHI

CHI '21: CHI Conference on Human Factors in Computing Systems

May 8 - 13, 2021

Yokohama, Japan

Acceptance Rates

Overall Acceptance Rate 6,164 of 23,696 submissions, 26%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
223
Total Downloads

Downloads (Last 12 months)43
Downloads (Last 6 weeks)9

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Anisha SSen ABain C(2024)Evaluating the Potential and Pitfalls of AI-Powered Conversational Agents as Humanlike Virtual Health Carers in the Remote Management of Noncommunicable Diseases: Scoping ReviewJournal of Medical Internet Research10.2196/5611426(e56114)Online publication date: 16-Jul-2024
https://doi.org/10.2196/56114
Ma YNordberg OZhang YRongve ABachinski MFjeld M(2024)Understanding Dementia Speech: Towards an Adaptive Voice Assistant for Enhanced CommunicationCompanion Proceedings of the 16th ACM SIGCHI Symposium on Engineering Interactive Computing Systems10.1145/3660515.3661326(15-21)Online publication date: 24-Jun-2024
https://dl.acm.org/doi/10.1145/3660515.3661326
Vogel CAhmad K(2023)Agreement and disagreement between major emotion recognition systemsKnowledge-Based Systems10.1016/j.knosys.2023.110759276:COnline publication date: 27-Sep-2023
https://dl.acm.org/doi/10.1016/j.knosys.2023.110759
Nath SDas UGhosh D(2023)A Religious Sentiment Detector Based on Machine Learning to Provide Meaningful Analysis of Religious TextsComputational Intelligence in Communications and Business Analytics10.1007/978-3-031-48876-4_13(165-184)Online publication date: 30-Nov-2023
https://doi.org/10.1007/978-3-031-48876-4_13
Ma YAbdelrahman YPetz BDrewes HAlt FHussmann HButz A(2022)Enthusiasts, Pragmatists, and Skeptics: Investigating Users’ Attitudes Towards Emotion- and Personality-Aware Voice Assistants across CulturesProceedings of Mensch und Computer 202210.1145/3543758.3543776(308-322)Online publication date: 4-Sep-2022
https://dl.acm.org/doi/10.1145/3543758.3543776

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents