abstract

A review of evaluation techniques for social dialogue systems

Authors:

ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

Pages 25 - 26

https://doi.org/10.1145/3139491.3139504

Published: 13 November 2017 Publication History

Get Access

Abstract

In contrast with goal-oriented dialogue, social dialogue has no clear measure of task success. Consequently, evaluation of these systems is notoriously hard. In this paper, we review current evaluation methods, focusing on automatic metrics. We conclude that turn-based metrics often ignore the context and do not account for the fact that several replies are valid, while end-of-dialogue rewards are mainly hand-crafted. Both lack grounding in human perceptions.

References

[1]

Helen Hastie. 2012. Metrics and evaluation of spoken dialogue systems. In Data-Driven Methods for Adaptive Spoken Dialogue Systems. Springer New York, 131–150.

Google Scholar

[2]

Anjuli Kannan and Oriol Vinyals. 2017. Adversarial Evaluation of Dialogue Models. CoRR abs/1701.08198 (2017). http://arxiv.org/abs/1701.08198

Google Scholar

[3]

Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep Reinforcement Learning for Dialogue Generation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP).

Crossref

Google Scholar

[4]

Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text summarization branches out: Proceedings of the ACL-04 workshop. Barcelona, Spain, 74–81. http://aclweb.org/anthology/W04-1013

Google Scholar

[5]

Chia-Wei Liu, Ryan Lowe, Iulian Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation. In Proceedings of Empirical Methods in Natural Language Processing (EMNLP). Austin, TX, USA. arXiv:1603.08023.

Crossref

Google Scholar

[6]

Ryan Lowe, Michael Noseworthy, Iulian Serban, Nicolas Angelard-Gontier, Yoshua Bengio, and Joelle Pineau. 2017. Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses. In (under review).

Google Scholar

[7]

Ryan Lowe, Iulian Vlad Serban, Michael Noseworthy, Laurent Charlin, and Joelle Pineau. 2016. On the Evaluation of Dialogue Systems with Next Utterance Classification. In Proceedings of the SIGDIAL 2016 Conference, The 17th Annual Meeting of the Special Interest Group on Discourse and Dialogue, 13-15 September 2016, Los Angeles, CA, USA. 264–269. http://aclweb.org/anthology/W/W16/W16-3634.pdf

Crossref

Google Scholar

[8]

Mike McTear. 2004. Spoken Dialogue Technology: Toward the Conversational User Interface. Springer, London.

Digital Library

Google Scholar

[9]

Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: A Method for Automatic Evaluation of Machine Translation. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) (ACL ’02). 311–318.

Digital Library

Google Scholar

[10]

Verena Rieser and Oliver Lemon. 2011. Reinforcement Learning for Adaptive Dialogue Systems: A Data-driven Methodology for Dialogue Management and Natural Language Generation. Book Series: Theory and Applications of Natural Language Processing, Springer, Berlin/Heidelberg.

Digital Library

Google Scholar

[11]

Alessandro Sordoni, Michel Galley, Michael Auli, Chris Brockett, Yangfeng Ji, Margaret Mitchell, Jian-Yun Nie, Jianfeng Gao, and Bill Dolan. 2015. A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In Proceedings of the 2015 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT). Denver, Colorado.

Crossref

Google Scholar

[12]

Oriol Vinyals and Quoc Le. 2015. A neural conversational model. arXiv preprint arXiv:1506.05869 (2015).

Google Scholar

[13]

Zhou Yu, Ziyu Xu, Alan W. Black, and Alexander I. Rudnicky. 2016. Strategy and Policy Learning for Non-Task-Oriented Conversational Systems. In Proceedings of the SIGDIAL. Abstract 1 Introduction 2 Automatic Metrics 2.1 Word-Overlap Metrics 2.2 Machine Learning Methods for Dialogue Evaluation 2.3 Reward-based Metrics 3 Conclusion and Discussion References

Google Scholar

Cited By

View all

Paetzel-Prüsmann MLehman JGomez CKennedy J(2024)An Automatic Evaluation Framework for Social Conversations with RobotsProceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction10.1145/3648536.3648543(56-64)Online publication date: 9-Mar-2024
https://dl.acm.org/doi/10.1145/3648536.3648543
Srivastava HSankar R(2023)Cooperative Attention-Based Learning between Diverse Data SourcesAlgorithms10.3390/a1605024016:5(240)Online publication date: 4-May-2023
https://doi.org/10.3390/a16050240
Kath HLüers BGouvêa TSonntag D(2023)Lost in Dialogue: A Review and Categorisation of Current Dialogue System Approaches and Technical SolutionsKI 2023: Advances in Artificial Intelligence10.1007/978-3-031-42608-7_9(98-113)Online publication date: 18-Sep-2023
https://doi.org/10.1007/978-3-031-42608-7_9
Show More Cited By

Index Terms

A review of evaluation techniques for social dialogue systems
1. General and reference
  1. Cross-computing tools and techniques
    1. Evaluation
  2. Document types
    1. Surveys and overviews
2. Human-centered computing
  1. Human computer interaction (HCI)
    1. Interaction paradigms
      1. Natural language interfaces

Recommendations

Survey on evaluation methods for dialogue systems
Abstract
In this paper, we survey the methods and concepts developed for the evaluation of dialogue systems. Evaluation, in and of itself, is a crucial part during the development process. Often, dialogue systems are evaluated by means of human evaluations ...
Dialog acts in greeting and leavetaking in social talk
ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

Conversation proceeds through dialogue moves or acts, and dialog act annotation can aid the design of artificial dialog. While many dialogs are task-based or instrumental, with clear goals, as in the case of a service encounter or business meeting, ...
Towards Conversationally Intelligent Dialog Systems
CHI EA '22: Extended Abstracts of the 2022 CHI Conference on Human Factors in Computing Systems

Spoken dialog systems, lacking the means to address the complex phenomena of spontaneous speech and conversational dynamics, force users into a constrained mode of dialog that resembles text-based interaction more closely than spoken conversation. Turn-...

Comments

Information & Contributors

Information

Published In

ISIAA 2017: Proceedings of the 1st ACM SIGCHI International Workshop on Investigating Social Interactions with Artificial Agents

November 2017

48 pages

ISBN:9781450355582

DOI:10.1145/3139491

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2017

Check for updates

Author Tags

Qualifiers

Abstract

Funding Sources

RAEng/Leverhulme Trust Senior Research Fellowship Scheme
EPSRC

Conference

ICMI '17

Sponsor:

SIGCHI

ICMI '17: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION

November 13, 2017

Glasgow, UK

Upcoming Conference

CHI '25

Sponsor:
sigchi

CHI Conference on Human Factors in Computing Systems

April 26 - May 1, 2025

Yokohama , Japan

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
243
Total Downloads

Downloads (Last 12 months)15
Downloads (Last 6 weeks)2

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Paetzel-Prüsmann MLehman JGomez CKennedy J(2024)An Automatic Evaluation Framework for Social Conversations with RobotsProceedings of the 2024 International Symposium on Technological Advances in Human-Robot Interaction10.1145/3648536.3648543(56-64)Online publication date: 9-Mar-2024
https://dl.acm.org/doi/10.1145/3648536.3648543
Srivastava HSankar R(2023)Cooperative Attention-Based Learning between Diverse Data SourcesAlgorithms10.3390/a1605024016:5(240)Online publication date: 4-May-2023
https://doi.org/10.3390/a16050240
Kath HLüers BGouvêa TSonntag D(2023)Lost in Dialogue: A Review and Categorisation of Current Dialogue System Approaches and Technical SolutionsKI 2023: Advances in Artificial Intelligence10.1007/978-3-031-42608-7_9(98-113)Online publication date: 18-Sep-2023
https://doi.org/10.1007/978-3-031-42608-7_9
Di Maro M(2021)Computational Grounding: An Overview of Common Ground Applications in Conversational AgentsItalian Journal of Computational Linguistics10.4000/ijcol.8907:1 | 2(133-156)Online publication date: 1-Dec-2021
https://doi.org/10.4000/ijcol.890
Gilmartin ESaam CTorres MSchlögl SClark LPorcheron M(2020)Pragmatics Research and Non-task Dialog TechnologyProceedings of the 2nd Conference on Conversational User Interfaces10.1145/3405755.3406142(1-3)Online publication date: 22-Jul-2020
https://dl.acm.org/doi/10.1145/3405755.3406142
Robb DLopes JPadilla SLaskov AChiyah Garcia FLiu XScharff Willners JValeyrie NLohan KLane DPatron PPetillot YChantler MHastie HHarrison SBardzell SNeustaedter CTatar D(2019)Exploring Interaction with Remote Autonomous Systems using Conversational AgentsProceedings of the 2019 on Designing Interactive Systems Conference10.1145/3322276.3322318(1543-1556)Online publication date: 18-Jun-2019
https://dl.acm.org/doi/10.1145/3322276.3322318

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Survey on evaluation methods for dialogue systems

Dialog acts in greeting and leavetaking in social talk

Towards Conversationally Intelligent Dialog Systems

Comments

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Survey on evaluation methods for dialogue systems

Dialog acts in greeting and leavetaking in social talk

Towards Conversationally Intelligent Dialog Systems

Comments

Information

Published In

Sponsors

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Funding Sources

Conference

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations