extended-abstract

HASI: A Model for Human-Agent Speech Interaction

Authors:

Thomas Mildner,

Kamyar Javanmardi,

Rainer MalakaAuthors Info & Claims

CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces

Article No.: 44, Pages 1 - 8

https://doi.org/10.1145/3640794.3665885

Published: 08 July 2024 Publication History

Abstract

In recent years, the widespread adoption of voice user interfaces (VUIs) highlighted the growing significance of speech interaction in our everyday activities. However, researchers, designers, and developers lack a dedicated interaction model tailored to the intricacies of communication with speech agents. This paper proposes a novel interaction model specifically crafted for speech interaction to better align with the evolving landscape of these systems. Incorporating traditional elements such as sender, message, and receiver, our model also integrates dynamic factors like context, user preferences, and evolving agent capabilities. Drawing from communication models and human-computer interaction (HCI) frameworks, this model aims to deepen our understanding of the process of human-agent speech interaction in real-world scenarios. By initiating discourse on a dedicated speech interaction model, this work serves as a basis for future exploration and refinement, adaptable to evolving technologies and user needs.

References

[1]

Gregory D Abowd and Russell Beale. 1991. Users, systems and interfaces: A unifying framework for interaction., 73–87 pages.

[2]

Dean C Barnlund. 2017. A transactional model of communication. In Communication theory. Routledge, Oxfordshire, England, UK, 47–57.

[3]

Michael Bonfert, Nima Zargham, Florian Saade, Robert Porzel, and Rainer Malaka. 2021. An Evaluation of Visual Embodiment for Voice Assistants on Smart Displays. In Proceedings of the 3rd Conference on Conversational User Interfaces (Bilbao (online), Spain) (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 16, 11 pages. https://doi.org/10.1145/3469595.3469611

Digital Library

[4]

MH Chignell and PA Hancock. 1988. Intelligent interface design. In Handbook of human-computer interaction. Elsevier, Amsterdam, The Netherlands, 969–995.

[5]

Christine Doran, John Aberdeen, Laurie Damianos, and Lynette Hirschman. 2003. Comparing several aspects of human-computer and human-human dialogues. In Current and new directions in discourse and dialogue. Springer, Berlin, Germany, 133–159.

[6]

Philip R. Doyle, Iona Gessinger, Justin Edwards, Leigh Clark, Odile Dumbleton, Diego Garaialde, Daniel Rough, Anna Bleakley, Holly P. Branigan, and Benjamin R. Cowan. 2023. The Partner Modelling Questionnaire: A validated self-report measure of perceptions toward machines as dialogue partners. arxiv:2308.07164 [cs.HC]

[7]

Richard Ellis and Ann McClintock. 1990. If you take my meaning: Theory into practice in human communication. Bloomsbury Academic, London, UK.

[8]

Sabah Al Fedaghi, Alaa Alsaqa, and Zahraa Fadel. 2009. Conceptual Model for Communication. arxiv:0912.0599 [cs.NI]

[9]

John Fiske. 2010. Introduction to communication studies. Routledge, Oxfordshire, UK.

[10]

David J. Gunkel. 2020. An introduction to communication and Artificial Intelligence. Polity Press, Medford, MA.

[11]

Cheryl M Hamilton. 2016. Communicating for success. Routledge, Oxfordshire, England, United Kingdom.

[12]

Razan Jaber, Sabrina Zhong, Sanna Kuoppamäki, Aida Hosseini, Iona Gessinger, Duncan P Brumby, Benjamin R. Cowan, and Donald Mcmillan. 2024. Cooking With Agents: Designing Context-aware Voice Interaction. In Proceedings of the CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA, ) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 551, 13 pages. https://doi.org/10.1145/3613904.3642183

Digital Library

[13]

Roman Jakobson. 1960. Linguistics and poetics. In Style in language. MA: MIT Press, Cambridge, Massachusetts, 350–377.

[14]

Richard G Jones. 2016. Communication in the Real World: An Introduction to Communication Studies. University of Minnesota Libraries Publishing, Minnesota, US.

[15]

Peter Kastberg. 2019. Knowledge Communication: Contours of a Research Agenda. Frank & Timme GmbH, Berlin, Germany.

[16]

Monica R Kimmel. 2020. A realist model of communication. Applications for informational technology and artificial cognitive systems. International Journal on Information Theory 9, 3/4 (2020), 1–16.

[17]

Raina Langevin, Ross J Lordon, Thi Avrahami, Benjamin R. Cowan, Tad Hirsch, and Gary Hsieh. 2021. Heuristic Evaluation of Conversational Agents. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (, Yokohama, Japan, ) (CHI ’21). Association for Computing Machinery, New York, NY, USA, Article 632, 15 pages. https://doi.org/10.1145/3411764.3445312

Digital Library

[18]

Stephen W Littlejohn and Karen A Foss. 2009. Encyclopedia of communication theory. Vol. 1. Sage, London, United Kingdom.

[19]

Sarah McDaid. 2009. A model for human-computer interaction based on human-human communication in a social context.Ph. D. Dissertation. London South Bank University.

[20]

Thomas Mildner, Orla Cooney, Anna-Maria Meck, Marion Bartl, Gian-Luca Savino, Philip R Doyle, Diego Garaialde, Leigh Clark, John Sloan, Nina Wenig, Rainer Malaka, and Jasmin Niess. 2024. Listening to the Voices: Describing Ethical Caveats of Conversational User Interfaces According to Experts and Frequent Users. In Proceedings of the CHI Conference on Human Factors in Computing Systems (CHI ’24). ACM, New York, NY, USA, Honolulu, HI, USA, 1–18. https://doi.org/10.1145/3613904.3642542

Digital Library

[21]

Kenneth J. Mitchell, Jessie B. Kennedy, and Peter J. Barclay. 1996. A framework for user-interfaces to databases. In Proceedings of the Workshop on Advanced Visual Interfaces (Gubbio, Italy) (AVI ’96). Association for Computing Machinery, New York, NY, USA, 81–90. https://doi.org/10.1145/948449.948462

Digital Library

[22]

Christine Murad, Cosmin Munteanu, Benjamin R. Cowan, and Leigh Clark. 2021. Finding a New Voice: Transitioning Designers from GUI to VUI Design. In Proceedings of the 3rd Conference on Conversational User Interfaces (Bilbao (online), Spain) (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 22, 12 pages. https://doi.org/10.1145/3469595.3469617

Digital Library

[23]

Uma Narula. 2006. Handbook of communication models, perspectives, strategies. Atlantic Publishers & Dist, Delhi, India.

[24]

Laurence Nigay. 1994. Conception et modélisation logicielles des systèmes interactifs: application aux interfaces multimodales. Ph. D. Dissertation. Université Joseph-Fourier-Grenoble I.

[25]

Laurence Nigay and Joëlle Coutaz. 1997. Multifeature systems: The care properties and their impact on software design.

[26]

Donald A Norman. 1988. The psychology of everyday things.Basic books, New York, US.

[27]

Paola R. Peña, Philip Doyle, Justin Edwards, Diego Garaialde, Daniel Rough, Anna Bleakley, Leigh Clark, Anita Tobar Henriquez, Holly Branigan, Iona Gessinger, and Benjamin R. Cowan. 2023. Audience design and egocentrism in reference production during human-computer dialogue. International Journal of Human-Computer Studies 176 (2023), 103058. https://doi.org/10.1016/j.ijhcs.2023.103058

Digital Library

[28]

Alisha Pradhan, Leah Findlater, and Amanda Lazar. 2019. " Phantom Friend" or" Just a Box with Information" Personification and Ontological Categorization of Smart Speaker-based Voice Assistants by Older Adults. Proceedings of the ACM on Human-Computer Interaction 3, CSCW (2019), 1–21.

[29]

Leon Reicherts, Nima Zargham, Michael Bonfert, Yvonne Rogers, and Rainer Malaka. 2021. May I Interrupt? Diverging Opinions On Proactive Smart Speakers. In Proceedings of the 3rd Conference on Conversational User Interfaces (Bilbao (online), Spain) (CUI ’21). Association for Computing Machinery, New York, NY, USA, Article 34, 10 pages. https://doi.org/10.1145/3469595.3469629

Digital Library

[30]

Malak Sadek, Rafael A Calvo, and Celine Mougenot. 2023. Trends, Challenges and Processes in Conversational Agent Design: Exploring Practitioners’ Views through Semi-Structured Interviews. In Proceedings of the 5th International Conference on Conversational User Interfaces (, Eindhoven, Netherlands, ) (CUI ’23). Association for Computing Machinery, New York, NY, USA, Article 13, 10 pages. https://doi.org/10.1145/3571884.3597143

Digital Library

[31]

Wilbur Schramm. 1997. The beginnings of communication study in America: A personal memoir. Sage, London, United Kingdom.

[32]

C. E. Shannon. 1948. A mathematical theory of communication. The Bell System Technical Journal 27, 3 (1948), 379–423. https://doi.org/10.1002/j.1538-7305.1948.tb01338.x

[33]

James Watson and Anne Hill. 2015. Dictionary of media and communication studies. Bloomsbury Publishing USA, New York, US.

[34]

Su-Fang Yeh, Meng-Hsin Wu, Tze-Yu Chen, Yen-Chun Lin, XiJing Chang, You-Hsuan Chiang, and Yung-Ju Chang. 2022. How to Guide Task-oriented Chatbot Users, and When: A Mixed-methods Study of Combinations of Chatbot Guidance Types and Timings. In CHI Conference on Human Factors in Computing Systems. ACM, New Orleans LA USA, 1–16. https://doi.org/10.1145/3491102.3501941

Digital Library

[35]

Nima Zargham, Michael Bonfert, Robert Porzel, Tanja Doring, and Rainer Malaka. 2022. Multi-Agent Voice Assistants: An Investigation of User Experience. In Proceedings of the 20th International Conference on Mobile and Ubiquitous Multimedia (Leuven, Belgium) (MUM ’21). Association for Computing Machinery, New York, NY, USA, 98–107. https://doi.org/10.1145/3490632.3490662

Digital Library

[36]

Nima Zargham, Mohamed Lamine Fetni, Laura Spillner, Thomas Muender, and Rainer Malaka. 2024. "I Know What You Mean": Context-Aware Recognition to Enhance Speech-Based Games. In Proceedings of the CHI Conference on Human Factors in Computing Systems (, Honolulu, HI, USA, ) (CHI ’24). Association for Computing Machinery, New York, NY, USA, Article 956, 18 pages. https://doi.org/10.1145/3613904.3642426

Digital Library

[37]

Nima Zargham, Johannes Pfau, Tobias Schnackenberg, and Rainer Malaka. 2022. “I Didn’t Catch That, But I’ll Try My Best”: Anticipatory Error Handling in a Voice Controlled Game. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (New Orleans, LA, USA) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 153, 13 pages. https://doi.org/10.1145/3491102.3502115

Digital Library

[38]

Nima Zargham, Leon Reicherts, Michael Bonfert, Sarah Theres Voelkel, Johannes Schoening, Rainer Malaka, and Yvonne Rogers. 2022. Understanding Circumstances for Desirable Proactive Behaviour of Voice Assistants: The Proactivity Dilemma. In Proceedings of the 4th Conference on Conversational User Interfaces (Glasgow, United Kingdom) (CUI ’22). Association for Computing Machinery, New York, NY, USA, Article 3, 14 pages. https://doi.org/10.1145/3543829.3543834

Digital Library

Cited By

Joshi RBengler K(2024)Crafting Human-AI Interaction: A Rhetorical Approach to Adaptive Interaction in Conversational AgentsProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688297(314-322)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688297

Index Terms

HASI: A Model for Human-Agent Speech Interaction
1. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Generative Model of Agent’s Behaviors in Human-Agent Interaction
ICMI '19: 2019 International Conference on Multimodal Interaction

A social interaction implies a social exchange between two or more persons, where they adapt and adjust their behaviors in response to their interaction partners. With the growing interest in human-agent interactions, it is desirable to make these ...
Effect of Speech Entrainment in Human-Computer Conversation: A Review
Intelligent Human Computer Interaction
Abstract
The phenomenon of entrainment in conversation is the process where participants become more similar to each other in terms of different verbal and non-verbal aspects such as acoustic-prosodic, lexical, syntactic, pitch, and speech rate. This ...
Voice in Human–Agent Interaction: A Survey

Social robots, conversational agents, voice assistants, and other embodied AI are increasingly a feature of everyday life. What connects these various types of intelligent agents is their ability to interact with people through voice. Voice is becoming ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CUI '24: Proceedings of the 6th ACM Conference on Conversational User Interfaces

July 2024

616 pages

ISBN:9798400705113

DOI:10.1145/3640794

Copyright © 2024 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Sponsors

SIGCHI: ACM Special Interest Group on Computer-Human Interaction

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 July 2024

Check for updates

Author Tags

Qualifiers

Extended-abstract
Research
Refereed limited

Conference

CUI '24

Sponsor:

SIGCHI

CUI '24: ACM Conversational User Interfaces 2024

July 8 - 10, 2024

Luxembourg, Luxembourg

Acceptance Rates

Overall Acceptance Rate 34 of 100 submissions, 34%

Upcoming Conference

CUI '25

Sponsor:
sigchi

ACM Conversational User Interfaces 2025

July 8 - 10, 2025

Waterloo , ON , Canada

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

1
Total Citations
View Citations
124
Total Downloads

Downloads (Last 12 months)124
Downloads (Last 6 weeks)38

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Joshi RBengler K(2024)Crafting Human-AI Interaction: A Rhetorical Approach to Adaptive Interaction in Conversational AgentsProceedings of the 12th International Conference on Human-Agent Interaction10.1145/3687272.3688297(314-322)Online publication date: 24-Nov-2024
https://dl.acm.org/doi/10.1145/3687272.3688297

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Figures

Tables

Media

View Table of Conten