research-article

Public Access

A Hybrid Retrieval-Generation Neural Conversation Model

Authors:

W. Bruce Croft,

Jingjing LiuAuthors Info & Claims

CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Pages 1341 - 1350

https://doi.org/10.1145/3357384.3357881

Published: 03 November 2019 Publication History

Abstract

Intelligent personal assistant systems that are able to have multi-turn conversations with human users are becoming increasingly popular. Most previous research has been focused on using either retrieval-based or generation-based methods to develop such systems. Retrieval-based methods have the advantage of returning fluent and informative responses with great diversity. However, the performance of the methods is limited by the size of the response repository. On the other hand, generation-based methods can produce highly coherent responses on any topics. But the generated responses are often generic and not informative due to the lack of grounding knowledge. In this paper, we propose a hybrid neural conversation model that combines the merits of both response retrieval and generation methods. Experimental results on Twitter and Foursquare data show that the proposed model outperforms both retrieval-based methods and generation-based methods (including a recently proposed knowledge-grounded neural conversation model) under both automatic evaluation metrics and human evaluation. We hope that the findings in this study provide new insights on how to integrate text retrieval and text generation models for building conversation systems.

References

[1]

D. Bahdanau, K. Cho, and Y. Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR abs/1409.0473 (2014).

[2]

A. Bordes, Y. Boureau, and J. Weston. 2017. Learning end-to-end goal-oriented Dialog. ICLR '17.

[3]

J. Chung, Ç. Gülçehre, K. Cho, and Y. Bengio. 2014. Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling. CoRR (2014).

[4]

B. Dhingra, L. Li, X. Li, J. Gao, Y. Chen, F. Ahmed, and L. Deng. 2017. Towards End-to-End Reinforcement Learning of Dialogue Agents for Information Access. In ACL'17.

[5]

J.L. Fleiss et al. 1971. Measuring Nominal Scale Agreement Among Many Raters. Psychological Bulletin 76, 5 (1971), 378--382.

[6]

M. Galley, C. Brockett, A. Sordoni, Y. Ji, M. Auli, C. Quirk, M. Mitchell, J. Gao, and B. Dolan. 2015. deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targets. CoRR abs/1506.06863 (2015).

[7]

J. Gao, M. Galley, and L. Li. 2018. Neural Approaches to Conversational AI. CoRR abs/1809.08267 (2018).

[8]

M. Ghazvininejad, C. Brockett, M. Chang, B. Dolan, J. Gao, W. Yih, and M. Galley. 2018. A Knowledge-Grounded Neural Conversation Model. In AAAI '18.

[9]

J. Guo, Y. Fan, Q. Ai, and W. B. Croft. 2016. A Deep Relevance Matching Model for Ad-hoc Retrieval. In CIKM '16.

[10]

J. Guo, Y. Fan, L. Pang, L. Yang, Q. Ai, H. Zamani, C. Wu, W. B. Croft, and X. Cheng. 2019. A Deep Look into Neural Ranking Models for Information Retrieval. CoRR (2019).

[11]

M. Henderson. 2015. Machine Learning for Dialog State Tracking: a Review.

[12]

S. Hochreiter and J. Schmidhuber. 1997. Long Short-Term Memory. Neural Comput. 9, 8 (Nov. 1997).

Digital Library

[13]

B. Hu, Z. Lu, H. Li, and Q. Chen. 2014. Convolutional Neural Network Architectures for Matching Natural Language Sentences. In NIPS '14.

Digital Library

[14]

P. Huang, X. He, J. Gao, L. Deng, A. Acero, and L. P. Heck. 2013. Learning Deep Structured Semantic Models for Web Search using Clickthrough Data. In CIKM '13.

[15]

Z. Ji, Z. Lu, and H. Li. 2014. An Information Retrieval Approach to Short Text Conversation. CoRR abs/1408.6988 (2014).

[16]

D. P. Kingma and J. Ba. 2014. Adam: A Method for Stochastic Optimization. CoRR (2014).

[17]

J. Li, M. Galley, C. Brockett, J. Gao, and B. Dolan. 2015. A Diversity-Promoting Objective Function for Neural Conversation Models. CoRR abs/1510.03055 (2015).

[18]

J. Li, M. Galley, C. Brockett, G. P. Spithourakis, J. Gao, and W. B. Dolan. 2016. A Persona-Based Neural Conversation Model. In ACL'16.

[19]

J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, and J. Gao. 2016. Deep Reinforcement Learning for Dialogue Generation. In EMNLP'16.

[20]

X. Liu, Y. Shen, K. Duh, and J. Gao. 2018. Stochastic Answer Networks for Machine Reading Comprehension. In ACL '18.

[21]

R. Lowe, N. Pow, I. Serban, and J. Pineau. 2015. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. CoRR abs/1506.08909 (2015).

[22]

T. Luong, H. Pham, and C. D. Manning. 2015. Effective Approaches to Attention based Neural Machine Translation. In EMNLP '15.

[23]

M. Mintz, S. Bills, R. Snow, and D. Jurafsky. 2009. Distant Supervision for Relation Extraction without Labeled Data. In ACL '09.

[24]

B. Mitra, F. Diaz, and N. Craswell. 2017. Learning to Match Using Local and Distributed Representations of Text for Web Search. In WWW '17.

[25]

G. Pandey, D. Contractor, V. Kumar, and S. Joshi. 2018. Exemplar Encoder-Decoder for Neural Conversation Generation. In ACL '18.

[26]

L. Pang, Y. Lan, J. Guo, J. Xu, S. Wan, and X. Cheng. 2016. Text Matching as Image Recognition. In AAAI '16.

Digital Library

[27]

Jay M. Ponte and W. B. Croft. 1998. A Language Modeling Approach to Information Retrieval. In SIGIR '98.

[28]

L. Qin, M. Galley, C. Brockett, X. Liu, X. Gao, B. Dolan, Y. Choi, and J. Gao. 2019. Conversing by Reading: Contentful Neural Conversation with On-demand Machine Reading. In ACL '19.

[29]

M. Qiu, F. Li, S. Wang, X. Gao, Y. Chen, W. Zhao, H. Chen, J. Huang, and W. Chu. 2017. AliMe Chat: A Sequence to Sequence and Rerank based Chatbot Engine. In ACL '17.

[30]

M. Qiu, L. Yang, F. Ji, W. Zhou, J. Huang, H. Chen, W. B. Croft, and W. Lin. 2018. Transfer Learning for Context-Aware Question Matching in Information-seeking Conversations in E-commerce. In ACL '18.

[31]

A. Ritter, C. Cherry, and W. B. Dolan. 2011. Data-Driven Response Generation in Social Media. In ACL '11.

Digital Library

[32]

S. Robertson and S. Walker. 1994. Some Simple Effective Approximations to the 2-Poisson Model for Probabilistic Weighted Retrieval. In SIGIR '94.

Digital Library

[33]

L. Shang, Z. Lu, and H. Li. 2015. Neural Responding Machine for Short-Text Conversation. In ACL '15.

[34]

Y. Song, C. Li, J. Nie, M. Zhang, D. Zhao, and R. Yan. 2018. An Ensemble of Retrieval-Based and Generation-Based Human-Computer Conversation Systems. In IJCAI '18.

[35]

A. Sordoni, M. Galley, M. Auli, C. Brockett, Y. Ji, M. Mitchell, J. Nie, J. Gao, and B. Dolan. 2015. A Neural Network Approach to Context-Sensitive Generation of Conversational Responses. In NAACL '15.

[36]

Z. Tian, R. Yan, L. Mou, Y. Song, Y. Feng, and D. Zhao. 2017. How to Make Context More Useful? An Empirical Study on Context-Aware Neural Conversational Models. In ACL '17.

[37]

O. Vinyals and Q. V. Le. 2015. A Neural Conversational Model. CoRR abs/1506.05869 (2015).

[38]

Y. Wu, F. Wei, S. Huang, Z. Li, and M. Zhou. 2018. Response Generation by Context-aware Prototype Editing. CoRR (2018).

[39]

Y.Wu,W.Wu, C. Xing, M. Zhou, and Z. Li. 2017. Sequential Matching Network: A New Architecture for Multi-turn Response Selection in Retrieval-Based Chatbots. In ACL '17.

[40]

C. Xiong, Z. Dai, J. Callan, Z. Liu, and R. Power. 2017. End-to-End Neural Ad-hoc Ranking with Kernel Pooling. In SIGIR '17.

[41]

R. Yan, Y. Song, and H. Wu. 2016. Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. In SIGIR.

[42]

R. Yan, Y. Song, X. Zhou, and H. Wu. 2016. "Shall I Be Your Chat Companion?": Towards an Online Human-Computer Conversation System. In CIKM '16.

[43]

R. Yan, D. Zhao, and W. E. 2017. Joint Learning of Response Ranking and Next Utterance Suggestion in Human-Computer Conversation System. In SIGIR '17.

[44]

L. Yang, Q. Ai, J. Guo, andW. B. Croft. 2016. aNMM: Ranking Short Answer Texts with Attention-Based Neural Matching Model. In CIKM '16.

Digital Library

[45]

L. Yang, M. Qiu, C. Qu, J. Guo, Y. Zhang,W. B. Croft, J. Huang, and H. Chen. 2018. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. In SIGIR '18.

[46]

L. Yang, H. Zamani, Y. Zhang, J. Guo, and W. B. Croft. 2017. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation. CoRR (2017).

[47]

J. Yu, M. Qiu, J. Jiang, J. Huang, S. Song, W. Chu, and H. Chen. 2018. Modelling Domain Relationships for Transfer Learning on Retrieval-based Question Answering Systems in E-commerce. WSDM '18.

[48]

R. Zhang, J. Guo, Y. Fan, Y. Lan, J. Xu, and X. Cheng. 2018. Learning to Control the Specificity in Neural Response Generation. In ACL '18.

[49]

Y. Zhang, M. Galley, J. Gao, Z. Gan, X. Li, C. Brockett, and B. Dolan. 2018. Generating Informative and Diverse Conversational Responses via Adversarial Information Maximization. CoRR (2018).

[50]

L. Zhou, J. Gao, D. Li, and H. Shum. 2018. The Design and Implementation of XiaoIce, an Empathetic Social Chatbot. CoRR (2018).

[51]

X. Zhou, D. Dong, H. Wu, S. Zhao, D. Yu, H. Tian, X. Liu, and R. Yan. 2016. Multi-view Response Selection for Human-Computer Conversation. In EMNLP.

Cited By

Song YZhao XWong RBaeza-Yates RBonchi F(2024)Marrying Dialogue Systems with Data Visualization: Interactive Data Visualization Generation from Natural Language ConversationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671935(2733-2744)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671935
Bénédict GZhang RMetzler DYates AJiang ZHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Gen-IR @ SIGIR 2024: The Second Workshop on Generative Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657982(3029-3032)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657982
Chen JLi SKan BWang FHao T(2024)Leveraging Intent Entity Enhancement for Task-Oriented Dialogue2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10649940(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10649940
Show More Cited By

Index Terms

A Hybrid Retrieval-Generation Neural Conversation Model
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Question answering

Recommendations

Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Intelligent personal assistant systems with either text-based or voice-based conversational interfaces are becoming increasingly popular around the world. Retrieval-based conversation models have the advantages of returning fluent and informative ...
Improve the effectiveness of the opinion retrieval and opinion polarity classification
CIKM '08: Proceedings of the 17th ACM conference on Information and knowledge management

Opinion retrieval is a document retrieving and ranking process. A relevant document must be relevant to the query and contain opinions toward the query. Opinion polarity classification is an extension of opinion retrieval. It classifies the retrieved ...
Opinion retrieval from blogs
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management

Opinion retrieval is a document retrieval process, which requires documents to be retrieved and ranked according to their opinions about a query topic. A relevant document must satisfy two criteria: relevant to the query topic, and contains opinions ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

November 2019

3373 pages

ISBN:9781450369763

DOI:10.1145/3357384

General Chairs:
Wenwu Zhu
Tsinghua University, China
,
Dacheng Tao
University of Massachusetts, USA
,
Xueqi Cheng
Institute of Computing Technology, CAS, China
,
Program Chairs:
Peng Cui
Tsinghua University, China
,
Elke Rundensteiner
Worcester Polytechnic Institute, USA
,
David Carmel
Amazon Research, USA
,
Qi He
LinkedIn, USA
,
Jeffrey Xu Yu
Chinese University of Hong Kong, China

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF

Conference

CIKM '19

Sponsor:

CIKM '19: The 28th ACM International Conference on Information and Knowledge Management

November 3 - 7, 2019

Beijing, China

Acceptance Rates

CIKM '19 Paper Acceptance Rate 202 of 1,031 submissions, 20%;

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

43
Total Citations
View Citations
1,278
Total Downloads

Downloads (Last 12 months)253
Downloads (Last 6 weeks)23

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Song YZhao XWong RBaeza-Yates RBonchi F(2024)Marrying Dialogue Systems with Data Visualization: Interactive Data Visualization Generation from Natural Language ConversationsProceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining10.1145/3637528.3671935(2733-2744)Online publication date: 25-Aug-2024
https://dl.acm.org/doi/10.1145/3637528.3671935
Bénédict GZhang RMetzler DYates AJiang ZHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Gen-IR @ SIGIR 2024: The Second Workshop on Generative Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657982(3029-3032)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657982
Chen JLi SKan BWang FHao T(2024)Leveraging Intent Entity Enhancement for Task-Oriented Dialogue2024 International Joint Conference on Neural Networks (IJCNN)10.1109/IJCNN60899.2024.10649940(1-8)Online publication date: 30-Jun-2024
https://doi.org/10.1109/IJCNN60899.2024.10649940
Canabal-Juanatey MAlonso-Moral JCatala ABugarín-Diz A(2024)Enriching interactive explanations with fuzzy temporal constraint networksInternational Journal of Approximate Reasoning10.1016/j.ijar.2024.109128171:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.ijar.2024.109128
Youness FElshenawy AMakour M(2024)Bidirectional attentional mechanism for Arabic chatbotInternational Journal of Information Technology10.1007/s41870-024-01777-216:5(3109-3120)Online publication date: 29-Mar-2024
https://doi.org/10.1007/s41870-024-01777-2
Varshney DEkbal ATiwari MNagaraja G(2023)EmoKbGAN: Emotion controlled response generation using Generative Adversarial Network for knowledge grounded conversationPLOS ONE10.1371/journal.pone.028045818:2(e0280458)Online publication date: 16-Feb-2023
https://doi.org/10.1371/journal.pone.0280458
Abdallah APiryani BJatowt A(2023)Exploring the state of the art in legal QA systemsJournal of Big Data10.1186/s40537-023-00802-810:1Online publication date: 12-Aug-2023
https://doi.org/10.1186/s40537-023-00802-8
Gonçalves GMagalhaes JCallan JYoshioka MKiseleva JAliannejadi M(2023)Conversational Search with Random Walks over Entity GraphsProceedings of the 2023 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3578337.3605125(77-85)Online publication date: 9-Aug-2023
https://dl.acm.org/doi/10.1145/3578337.3605125
Bénédict GZhang RMetzler DChen HDuh WHuang HKato MMothe JPoblete B(2023)Gen-IR@SIGIR 2023: The First Workshop on Generative Information RetrievalProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591923(3460-3463)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591923
Ling YCai FLiu JChen Hde Rijke M(2023)Keep and Select: Improving Hierarchical Context Modeling for Multi-Turn Response GenerationIEEE Transactions on Neural Networks and Learning Systems10.1109/TNNLS.2021.311270034:7(3636-3649)Online publication date: Jul-2023
https://doi.org/10.1109/TNNLS.2021.3112700
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents