research-article

User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems

Authors:

Helen MengAuthors Info & Claims

WWW '22: Proceedings of the ACM Web Conference 2022

Pages 2998 - 3008

https://doi.org/10.1145/3485447.3512020

Published: 25 April 2022 Publication History

Abstract

User Satisfaction Estimation (USE) is an important yet challenging task in goal-oriented conversational systems. Whether the user is satisfied with the system largely depends on the fulfillment of the user’s needs, which can be implicitly reflected by users’ dialogue acts. However, existing studies often neglect the sequential transitions of dialogue act or rely heavily on annotated dialogue act labels when utilizing dialogue acts to facilitate USE. In this paper, we propose a novel framework, namely USDA, to incorporate the sequential dynamics of dialogue acts for predicting user satisfaction, by jointly learning User Satisfaction Estimation and Dialogue Act Recognition tasks. In specific, we first employ a Hierarchical Transformer to encode the whole dialogue context, with two task-adaptive pre-training strategies to be a second-phase in-domain pre-training for enhancing the dialogue modeling ability. In terms of the availability of dialogue act labels, we further develop two variants of USDA to capture the dialogue act information in either supervised or unsupervised manners. Finally, USDA leverages the sequential transitions of both content and act features in the dialogue to predict the user satisfaction. Experimental results on four benchmark goal-oriented dialogue datasets across different applications show that the proposed method substantially and consistently outperforms existing methods on USE, and validate the important role of dialogue act sequences in USE.

References

[1]

Ali Ahmadvand, Jason Ingyu Choi, and Eugene Agichtein. 2019. Contextual Dialogue Act Classification for Open-Domain Conversational Agents. In SIGIR 2019. 1273–1276.

[2]

Praveen Kumar Bodigutla, Aditya Tiwari, Spyros Matsoukas, Josep Valls-Vargas, and Lazaros Polymenakos. 2020. Joint Turn and Dialogue level User Satisfaction Estimation on Mulit-Domain Conversations. In Findings, EMNLP 2020. 3897–3909.

[3]

Deng Cai, Yizhe Zhang, Yichen Huang, Wai Lam, and Bill Dolan. 2020. Narrative Incoherence Detection. CoRR abs/2012.11157(2020).

[4]

Wanling Cai and Li Chen. 2020. Predicting User Intents and Satisfaction with Dialogue-based Conversational Recommendations. In UMAP 2020. 33–42.

[5]

Christophe Cerisara, Somayeh Jafaritazehjani, Adedayo Oluokun, and Hoa T. Le. 2018. Multi-task dialog act and sentiment recognition on Mastodon. In COLING 2018. 745–754.

[6]

Hongshen Chen, Xiaorui Liu, Dawei Yin, and Jiliang Tang. 2017. A Survey on Dialogue Systems: Recent Advances and New Frontiers. SIGKDD Explor. 19, 2 (2017), 25–35.

Digital Library

[7]

Meng Chen, Ruixue Liu, Lei Shen, Shaozu Yuan, Jingyan Zhou, Youzheng Wu, Xiaodong He, and Bowen Zhou. 2020. The JDDC Corpus: A Large-Scale Multi-Turn Chinese Dialogue Dataset for E-commerce Customer Service. In LREC 2020. 459–466.

[8]

Zheqian Chen, Rongqin Yang, Zhou Zhao, Deng Cai, and Xiaofei He. 2018. Dialogue Act Recognition via CRF-Attentive Structured Network. In SIGIR 2018. 225–234.

Digital Library

[9]

Jackie Chi Kit Cheung and Xiao Li. 2012. Sequence clustering and labeling for unsupervised query intent discovery. In WSDM 2012. 383–392.

Digital Library

[10]

Jason Ingyu Choi, Ali Ahmadvand, and Eugene Agichtein. 2019. Offline and Online Satisfaction Prediction in Open-Domain Conversational Systems. In CIKM 2019. 1281–1290.

[11]

Jeffrey Dalton, Chenyan Xiong, Vaibhav Kumar, and Jamie Callan. 2020. CAsT-19: A Dataset for Conversational Information Seeking. In SIGIR 2020. 1985–1988.

Digital Library

[12]

Yang Deng, Yaliang Li, Fei Sun, Bolin Ding, and Wai Lam. 2021. Unified Conversational Recommendation Policy Learning via Graph-based Reinforcement Learning. In SIGIR. 1431–1441.

[13]

Yang Deng, Yaliang Li, Wenxuan Zhang, Bolin Ding, and Wai Lam. 2022. Towards Personalized Answer Generation in E-Commerce via Multi-Perspective Preference Modeling. ACM Trans. Inf. Syst.(2022).

[14]

Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Intra-/Inter-Interaction Network with Latent Interaction Modeling for Multi-turn Response Selection. In COLING. 4981–4992.

[15]

Yang Deng, Wenxuan Zhang, and Wai Lam. 2020. Opinion-aware Answer Generation for Review-driven Question Answering in E-Commerce. In CIKM.

[16]

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT 2019. 4171–4186.

[17]

Klaus-Peter Engelbrecht, Florian Gödde, Felix Hartard, Hamed Ketabdar, and Sebastian Möller. 2009. Modeling User Satisfaction with Hidden Markov Models. In SIGDIAL 2009. 170–177.

[18]

Mihail Eric, Rahul Goel, Shachi Paul, Abhishek Sethi, Sanchit Agarwal, Shuyang Gao, Adarsh Kumar, Anuj Kumar Goyal, Peter Ku, and Dilek Hakkani-Tür. 2020. MultiWOZ 2.1: A Consolidated Multi-Domain Dialogue Dataset with State Corrections and State Tracking Baselines. In LREC 2020. 422–428.

[19]

Liyi Guo, Rui Lu, Haoqi Zhang, Junqi Jin, Zhenzhe Zheng, Fan Wu, Jin Li, Haiyang Xu, Han Li, Wenkai Lu, Jian Xu, and Kun Gai. 2020. A Deep Prediction Network for Understanding Advertiser Intent and Satisfaction. In CIKM 2020. 2501–2508.

Digital Library

[20]

Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks. In ACL 2020. 8342–8360.

[21]

Sunao Hara, Norihide Kitaoka, and Kazuya Takeda. 2010. Estimation Method of User Satisfaction Using N-gram-based Dialog History Model for Spoken Dialog System. In LREC 2010.

[22]

Seyyed Hadi Hashemi, Kyle Williams, Ahmed El Kholy, Imed Zitouni, and Paul A. Crook. 2018. Measuring User Satisfaction on Smart Speaker Intelligent Assistants Using Intent Sensitive Query Embeddings. In CIKM 2018. 1183–1192.

[23]

Pan Ji, Tong Zhang, Hongdong Li, Mathieu Salzmann, and Ian D. Reid. 2017. Deep Subspace Clustering Networks. In NeurIPS 2017. 24–33.

[24]

Jiepu Jiang, Ahmed Hassan Awadallah, Xiaolin Shi, and Ryen W. White. 2015. Understanding and Predicting Graded Search Satisfaction. In WSDM 2015. 57–66.

[25]

Wenxiang Jiao, Haiqin Yang, Irwin King, and Michael R. Lyu. 2019. HiGRU: Hierarchical Gated Recurrent Units for Utterance-Level Emotion Recognition. In NAACL-HLT 2019. 397–406.

[26]

Mohammad Kachuee, Hao Yuan, Young-Bum Kim, and Sungjin Lee. 2021. Self-Supervised Contrastive Learning for Efficient User Satisfaction Prediction in Conversational Agents. In NAACL-HLT 2021. 4053–4064.

[27]

Diane Kelly. 2009. Methods for Evaluating Interactive Information Retrieval Systems with Users. Found. Trends Inf. Retr. 3, 1-2 (2009), 1–224.

Digital Library

[28]

Julia Kiseleva, Kyle Williams, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Predicting User Satisfaction with Intelligent Assistants. In SIGIR 2016. 45–54.

[29]

Julia Kiseleva, Kyle Williams, Jiepu Jiang, Ahmed Hassan Awadallah, Aidan C. Crook, Imed Zitouni, and Tasos Anastasakos. 2016. Understanding User Satisfaction with Intelligent Assistants. In CHIIR 2016. 121–130.

[30]

Harshit Kumar, Arvind Agarwal, and Sachindra Joshi. 2019. A Practical Dialogue-Act-Driven Conversation Model for Multi-Turn Response Selection. In EMNLP-IJCNLP 2019. 1980–1989.

[31]

John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira. 2001. Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data. In ICML 2001. 282–289.

Digital Library

[32]

Wenqiang Lei, Xiangnan He, Yisong Miao, Qingyun Wu, Richang Hong, Min-Yen Kan, and Tat-Seng Chua. 2020. Estimation-Action-Reflection: Towards Deep Interaction Between Conversational and Recommender Systems. In WSDM 2020. 304–312.

[33]

Wenqiang Lei, Xisen Jin, Min-Yen Kan, Zhaochun Ren, Xiangnan He, and Dawei Yin. 2018. Sequicity: Simplifying Task-oriented Dialogue Systems with Single Sequence-to-Sequence Architectures. In ACL 2018. 1437–1447.

[34]

Raymond Li, Samira Ebrahimi Kahou, Hannes Schulz, Vincent Michalski, Laurent Charlin, and Chris Pal. 2018. Towards Deep Conversational Recommendations. In NeurIPS 2018. 9748–9758.

[35]

Jiawei Liu, Zhe Gao, Yangyang Kang, Zhuoren Jiang, Guoxiu He, Changlong Sun, Xiaozhong Liu, and Wei Lu. 2021. Time to Transfer: Predicting and Evaluating Machine-Human Chatting Handoff. In AAAI 2021. 5841–5849.

[36]

Rishabh Mehrotra, Mounia Lalmas, Doug Kenney, Thomas Lim-Meng, and Golli Hashemian. 2019. Jointly Leveraging Intent and Interaction Signals to Predict User Satisfaction with Slate Recommendations. In WWW 2019. 1256–1267.

[37]

Rishabh Mehrotra, Imed Zitouni, Ahmed Hassan Awadallah, Ahmed El Kholy, and Madian Khabsa. 2017. User Interaction Sequences for Search Satisfaction Prediction. In SIGIR 2017. 165–174.

[38]

Mohsen Mesgar, Sebastian Bücker, and Iryna Gurevych. 2020. Dialogue Coherence Assessment Without Explicit Dialogue Act Labels. In ACL 2020. 1439–1450.

[39]

Hugh Perkins and Yi Yang. 2019. Dialog Intent Induction with Deep Multi-View Clustering. In EMNLP-IJCNLP 2019. 4014–4023.

[40]

Libo Qin, Zhouyang Li, Wanxiang Che, Minheng Ni, and Ting Liu. 2021. Co-GAT: A Co-Interactive Graph Attention Network for Joint Dialog Act Recognition and Sentiment Classification. In AAAI 2021. 13709–13717.

[41]

Chen Qu, Liu Yang, W. Bruce Croft, Johanne R. Trippas, Yongfeng Zhang, and Minghui Qiu. 2018. Analyzing and Characterizing User Intent in Information-seeking Conversations. In SIGIR 2018. 989–992.

[42]

Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. 2020. Towards Scalable Multi-Domain Conversational Agents: The Schema-Guided Dialogue Dataset. In AAAI 2020. 8689–8696.

[43]

Pengjie Ren, Zhongkun Liu, Xiaomeng Song, Hongtao Tian, Zhumin Chen, Zhaochun Ren, and Maarten de Rijke. 2021. Wizard of Search Engine: Access to Information Through Conversations with Search Engines. In SIGIR 2021.

Digital Library

[44]

Abigail See and Christopher D. Manning. 2021. Understanding and predicting user dissatisfaction in a neural generative chatbot. In SIGdial. 1–12.

[45]

Kaisong Song, Lidong Bing, Wei Gao, Jun Lin, Lujun Zhao, Jiancheng Wang, Changlong Sun, Xiaozhong Liu, and Qi Zhang. 2019. Using Customer Service Dialogues for Satisfaction Analysis with Context-Assisted Multiple Instance Learning. In EMNLP-IJCNLP 2019. 198–207.

[46]

Andreas Stolcke, Klaus Ries, Noah Coccaro, Elizabeth Shriberg, Rebecca Bates, Daniel Jurafsky, Paul Taylor, Rachel Martin, Carol Van Ess-Dykema, and Marie Meteer. 2000. Dialogue act modeling for automatic tagging and recognition of conversational speech. Computational linguistics 26, 3 (2000), 339–373.

[47]

Ning Su, Jiyin He, Yiqun Liu, Min Zhang, and Shaoping Ma. 2018. User Intent, Behaviour, and Perceived Satisfaction in Product Search. In WSDM 2018. 547–555.

Digital Library

[48]

Weiwei Sun, Shuo Zhang, Krisztian Balog, Zhaochun Ren, Pengjie Ren, Zhumin Chen, and Maarten de Rijke. 2021. Simulating User Satisfaction for the Evaluation of Task-oriented Dialogue Systems. In SIGIR 2021.

[49]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In NeurIPS 2017. 5998–6008.

Digital Library

[50]

Kai Wang, Junfeng Tian, Rui Wang, Xiaojun Quan, and Jianxing Yu. 2020. Multi-Domain Dialogue Acts and Response Co-Generation. In ACL 2020. 7125–7134.

[51]

Zhijing Wu, Yiqun Liu, Qianfan Zhang, Kailu Wu, Min Zhang, and Shaoping Ma. 2019. The Influence of Image Search Intents on User Behavior and Satisfaction. In WSDM 2019. 645–653.

[52]

Liu Yang, Minghui Qiu, Chen Qu, Cen Chen, Jiafeng Guo, Yongfeng Zhang, W. Bruce Croft, and Haiqing Chen. 2020. IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems. In WWW 2020. 2592–2598.

Digital Library

[53]

Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alexander J. Smola, and Eduard H. Hovy. 2016. Hierarchical Attention Networks for Document Classification. In NAACL-HLT 2016. 1480–1489.

[54]

Yue Yu, Siyao Peng, and Grace Hui Yang. 2019. Modeling Long-Range Context for Concurrent Dialogue Acts Recognition. In CIKM 2019. 2277–2280.

[55]

Zhaohao Zeng, Sosuke Kato, Tetsuya Sakai, and Inho Kang. 2020. Overview of the NTCIR-15 Dialogue Evaluation (DialEval-1) Task. In NTCIR–15. 13–34.

Cited By

Meem JRashid MHristidis V(2024)Modeling the impact of out-of-schema questions in task-oriented dialog systemsData Mining and Knowledge Discovery10.1007/s10618-024-01039-638:4(2466-2494)Online publication date: 4-Jun-2024
https://doi.org/10.1007/s10618-024-01039-6
Mazumder SLiu BMazumder SLiu B(2024)Continual Learning in Chit-Chat SystemsLifelong and Continual Learning Dialogue Systems10.1007/978-3-031-48189-5_5(103-126)Online publication date: 9-Jan-2024
https://doi.org/10.1007/978-3-031-48189-5_5
Hu ZFeng YLuu AHooi BLipani AFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulators to Enhance Dialogue SystemProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615220(3953-3957)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615220
Show More Cited By

Index Terms

User Satisfaction Estimation with Sequential Dialogue Act Modeling in Goal-oriented Conversational Systems
1. Computing methodologies
2. Human-centered computing
  1. Human computer interaction (HCI)

Index terms have been assigned to the content through auto-classification.

Recommendations

Multimodal User Satisfaction Recognition for Non-task Oriented Dialogue Systems
ICMI '21: Proceedings of the 2021 International Conference on Multimodal Interaction

Multimodal dialogue systems (MDSs) are needed to allow users to converse with virtual agents that use natural language by sensing the multimodal behavior of users. One crucial step in the development of an MDS is measuring how well the dialogue system ...
Contextual Dialogue Act Classification for Open-Domain Conversational Agents
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Classifying the general intent of the user utterance in a conversation, also known as Dialogue Act (DA), e.g., open-ended question, statement of opinion, or request for an opinion, is a key step in Natural Language Understanding (NLU) for conversational ...
Dialogue Act Recognition via CRF-Attentive Structured Network
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Dialogue Act Recognition (DAR) is a challenging problem in dialogue interpretation, which aims to associate semantic labels to utterances and characterize the speaker's intention. Currently, many existing approaches formulate the DAR problem ranging ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

WWW '22: Proceedings of the ACM Web Conference 2022

April 2022

3764 pages

ISBN:9781450390965

DOI:10.1145/3485447

Editors:
Frédérique Laforest
INSA Lyon, France
,
Raphaël Troncy
EURECOM, France
,
Elena Simperl
King’s College London, UK
,
Deepak Agarwal
Pinterest, USA
,
Aristides Gionis
KTH Royal Institute of Technology, Sweden
,
Ivan Herman
W3C / retired
,
Lionel Médini
Université Lyon 1, France

Copyright © 2022 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGWEB: ACM Special Interest Group on Hypertext, Hypermedia, and Web

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 April 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Center for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission's InnoHK scheme

Conference

WWW '22

Sponsor:

SIGWEB

WWW '22: The ACM Web Conference 2022

April 25 - 29, 2022

Virtual Event, Lyon, France

Acceptance Rates

Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
401
Total Downloads

Downloads (Last 12 months)110
Downloads (Last 6 weeks)5

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Meem JRashid MHristidis V(2024)Modeling the impact of out-of-schema questions in task-oriented dialog systemsData Mining and Knowledge Discovery10.1007/s10618-024-01039-638:4(2466-2494)Online publication date: 4-Jun-2024
https://doi.org/10.1007/s10618-024-01039-6
Mazumder SLiu BMazumder SLiu B(2024)Continual Learning in Chit-Chat SystemsLifelong and Continual Learning Dialogue Systems10.1007/978-3-031-48189-5_5(103-126)Online publication date: 9-Jan-2024
https://doi.org/10.1007/978-3-031-48189-5_5
Hu ZFeng YLuu AHooi BLipani AFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)Unlocking the Potential of User Feedback: Leveraging Large Language Model as User Simulators to Enhance Dialogue SystemProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615220(3953-3957)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615220
Meng CAliannejadi Mde Rijke MFrommholz IHopfgartner FLee MOakes MLalmas MZhang MSantos R(2023)System Initiative Prediction for Multi-turn Conversational Information SeekingProceedings of the 32nd ACM International Conference on Information and Knowledge Management10.1145/3583780.3615070(1807-1817)Online publication date: 21-Oct-2023
https://dl.acm.org/doi/10.1145/3583780.3615070
Lin HFeng SGeishauser CLubis Nvan Niekerk CHeck MRuppik BVukovic RGasić MChen HDuh WHuang HKato MMothe JPoblete B(2023)EmoUS: Simulating User Emotions in Task-Oriented DialoguesProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592092(2526-2531)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3592092

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Table of Contents