Abstract
Chit-chat dialogue systems, also known as open-domain dialogue systems, focus on carrying out chit-chat type of conversations with users on any topic without specific goals to complete (see Sect. 1.1 for more details). The need to support such free-flow conversations often makes it challenging to build dialogue systems that can perform well in practice. Existing methods for building chit-chat systems mainly rely on collecting a dialogue corpus with context and response pairs and training a response generation model. However, no matter how big a corpus is collected for the training, it will always be limited in terms of the knowledge that the chatbot needs to successfully model relevant responses in post-deployment real-world conversations with users. The ability to continually learn from post-deployment conversation experiences and self-improve the response generation performance thus become essential for the success of commercial chit-chat systems. This chapter focuses on the topic.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
R. Aljundi, L. Caccia, E. Belilovsky, M. Caccia, M. Lin, L. Charlin, T. Tuytelaars, Online continual learning with maximally interfered retrieval, in Advances in Neural Information Processing Systems (2019)
S. Bao, H. He, F. Wang, W. Hua, H. Wang, W. Wenquan, W. Zhihua, Z. Guo, L. Hua, X. Huang, et al., Plato-xl: Exploring the large-scale pre-training of dialogue generation, in Findings of the Association for Computational Linguistics: AACL-IJCNLP (2022), pp. 107–118
E.M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the dangers of stochastic parrots: can language models be too big? in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610–623
R. Bommasani, D.A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M.S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models (2021). arXiv:2108.07258
Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), pp. 3667–3684
T. Brown, B. Mann, N. Ryder, M. Subbiah, J.D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., Language models are few-shot learners, in Advances in Neural Information Processing Systems 33 (2020), pp. 1877–1901
A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P.K. Dokania, P.H.S. Torr, M. Ranzato, Continual learning with tiny episodic memories, in Workshop on Multi-Task and Lifelong Reinforcement Learning (2019)
H. Chen, X. Liu, D. Yin, J. Tang, A survey on dialogue systems: recent advances and new frontiers. ACM SIGKDD Explorations Newsl 19(2), 25–35 (2017)
G. Chen, X. Li, S. Xiao, C. Zhang, L. Xianghua, Racl: a robust adaptive contrastive learning method for conversational satisfaction prediction. Pattern Recognit. 138, 109386 (2023)
J.I. Choi, E. Agichtein, Quantifying the effects of prosody modulation on user engagement and satisfaction in conversational systems, in Proceedings of the 2020 Conference on Human Information Interaction and Retrieval (2020), pp. 417–421
J.I. Choi, A. Ahmadvand, E. Agichtein, Offline and online satisfaction prediction in open-domain conversational systems, in Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), pp. 1281–1290
P.F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, Deep reinforcement learning from human preferences, in Advances in Neural Information Processing Systems 30 (2017)
W.H. DeLone, E.R. McLean, Information systems success: the quest for the dependent variable. Inf. Syst. Res. 3(1), 60–95 (1992)
Y. Deng, W. Zhang, W. Lam, H. Cheng, H. Meng, User satisfaction estimation with sequential dialogue act modeling in goal-oriented conversational systems, in Proceedings of the ACM Web Conference (2022), pp. 2998–3008
R. Gabriel, Y. Liu, A. Gottardi, M. Eric, A. Khatri, A. Chadha, Q. Chen, B. Hedayatnia, P. Rajan, A. Binici, et al., Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge (2019)
S. Gehman, S. Gururangan, M. Sap, Y. Choi, N.A. Smith, Realtoxicityprompts: evaluating neural toxic degeneration in language models, in Findings of the Association for Computational Linguistics: EMNLP (2020), pp. 3356–3369
Y. Guo, B. Liu, D. Zhao, Online continual learning through mutual information maximization, in International Conference on Machine Learning (PMLR, 2022), pp. 8109–8126
S.H. Hashemi, K. Williams, A. El Kholy, I. Zitouni, P.A. Crook, Measuring user satisfaction on smart speaker intelligent assistants using intent sensitive query embeddings, in Proceedings of the 27th ACM International Conference on Information and Knowledge Management (2018), pp. 1183–1192
S. Humeau, K. Shuster, M.-A. Lachaux, J. Weston, Poly-encoders: transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring, in International Conference on Learning Representations (2019)
J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with gpus. IEEE Trans. Big Data 7(3), 535–547 (2019)
D. Ju, J. Xu, Y.-L. Boureau, J. Weston, Learning from data in the mixed adversarial non-adversarial case: finding the helpers and ignoring the trolls (2022). arXiv:2208.03295
Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, G. Irving, Alignment of language agents (2021). arXiv:2103.14659
J. Kiseleva, K. Williams, J. Jiang, A.H. Awadallah, A.C. Crook, I. Zitouni, T. Anastasakos, Understanding user satisfaction with intelligent assistants, in Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (2016), pp. 121–130
M. Komeili, K. Shuster, J. Weston, Internet-augmented dialogue generation, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2022), pp. 8460–8478
J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, S. Legg, Scalable agent alignment via reward modeling: a research direction (2018). arXiv:1811.07871
J. Li, M. Galley, C. Brockett, J. Gao, W.B. Dolan, A diversity-promoting objective function for neural conversation models, in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016a), pp. 110–119
J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, J. Gao, Deep reinforcement learning for dialogue generation, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016b), pp. 1192–1202
R. Liang, R. Takanobu, F.-L. Li, J. Zhang, H. Chen, M. Huang, Turn-level user satisfaction estimation in e-commerce customer service, in Proceedings of the 4th Workshop on e-Commerce and NLP (2021), pp. 26–32
A. Lipani, B. Carterette, E. Yilmaz, How am i doing?: Evaluating conversational search systems offline. ACM Trans. Inf. Syst. (TOIS) 39(4), 1–22 (2021)
J. Lopes, How generic can dialogue breakdown detection be? the kth entry to dbdc3, in Proceedings of Dialog System Technology Challenge 6 (2017)
H. Lu, S. Bao, H. He, F. Wang, H. Wu, H. Wang, Towards boosting the open-domain chatbot with human feedback (2022). arXiv:2208.14165
H. Mei, M. Bansal, M. Walter, Coherent dialogue with attention-based language models, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)
OpenAI, Gpt-4 technical report (2023). arXiv:2303.08774 [cs.CL]
L. Ouyang, W. Jeffrey, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., Training language models to follow instructions with human feedback, in Advances in Neural Information Processing Systems 35 (2022), pp. 27730–27744
G. Pandey, D. Contractor, V. Kumar, S. Joshi, Exemplar encoder-decoder for neural conversation generation, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2018), pp. 1329–1338
A. Paranjape, A. See, K. Kenealy, H. Li, A. Hardy, P. Qi, K.R. Sadagopan, N.M. Phu, D. Soylu, C.D. Manning, Neural generation meets real people: towards emotionally engaging mixed-initiative conversations, in 3rd Proceedings of Alexa Prize (Alexa Prize 2019) (2020)
A. Ram, R. Prasad, C. Khatri, A. Venkatesh, R. Gabriel, Q. Liu, J. Nunn, B. Hedayatnia, M. Cheng, A. Nagar, et al., Conversational ai: the science behind the alexa prize (2018)
S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, E.M. Smith, Y.-L. Boureau, et al., Recipes for building an open-domain chatbot, in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (2021), pp. 300–325
H. Ryuichiro, K. Funakoshi, M. Inaba, Y. Tsunomori, T. Takahashi, K. Nobuhiro, Overview of dialogue breakdown detection challenge 3, in Proceedings of Dialogue System Technology Challenge (2017), p. 14
T. Sandbank, M. Shmueli-Scheuer, J. Herzig, D. Konopnicki, J. Richards, D. Piorkowski, Detecting egregious conversations between customers and virtual agents, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (2018), pp. 1802–1811
J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms (2017). arXiv:1707.06347
A. See, C.D. Manning, Understanding and predicting user dissatisfaction in a neural generative chatbot, in Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (2021), pp. 1–12
I.V. Serban, A. Sordoni, Y. Bengio, A. Courville, J. Pineau, Building end-to-end dialogue systems using generative hierarchical neural network models, in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (2016), pp. 3776–3783
I.V. Serban, R. Lowe, P. Henderson, L. Charlin, J. Pineau, A survey of available corpora for building data-driven dialogue systems: The journal version. Dialogue & Discourse 9(1), 1–49 (2018)
L. Shang, Z. Lu, H. Li, Neural responding machine for short-text conversation, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (2015), pp. 1577–1586
X. Shen, H. Su, Y. Li, W. Li, S. Niu, Y. Zhao, A. Aizawa, G. Long, A conditional variational framework for dialog generation, in 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017 (Association for Computational Linguistics (ACL), 2017), pp. 504–509
X. Shen, H. Su, S. Niu, V. Demberg, Improving variational encoder-decoders in dialogue generation, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
W. Shi, E. Dinan, K. Shuster, J. Weston, J. Xu, When life gives you lemons, make cherryade: converting feedback from bad responses into good labels (2022). arXiv:2210.15893
K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, J. Weston, Language models that seek for knowledge: modular search & generation for dialogue and prompt completion, in Findings of the Association for Computational Linguistics: EMNLP 2022 (2022)
K. Shuster, J. Urbanek, E. Dinan, A. Szlam, J. Weston, Dialogue in the wild: learning from a deployed role-playing game with humans and bots, in Findings of the Association for Computational Linguistics: ACL-IJCNLP (2021), pp. 611–624
K. Shuster, J. Xu, M. Komeili, D. Ju, E.M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane, et al., Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage (2022). arXiv:2208.03188
S. Steidl, C. Hacker, C. Ruff, A. Batliner, E. Nöth, J. Haas, Looking at the last two turns, i’d say this dialogue is doomed–measuring dialogue success, in Text, Speech and Dialogue: 7th International Conference, TSD 2004, Brno, Czech Republic, September 8–11, 2004. Proceedings 7 (Springer, 2004), pp. 629–636
N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, P.F. Christiano, Learning to summarize with human feedback, in Advances in Neural Information Processing Systems 33 (2020), pp. 3008–3021
Y. Sun, L. Wu, S. Song, X. Yu, X. He, G. Fu, Tracking satisfaction states for customer satisfaction prediction in e-commerce service chatbots, in Proceedings of the 29th International Conference on Computational Linguistics (2022), pp. 616–625
A. Tamkin, M. Brundage, J. Clark, D. Ganguli, Understanding the capabilities, limitations, and societal impact of large language models (2021). arXiv:2102.02503
J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rocktäschel, D. Kiela, A. Szlam, J. Weston, Learning to speak and act in a fantasy text adventure game, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2010) (2019), pp. 673–683
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems 30 (2017)
A. Venkatesh, C. Khatri, A. Ram, F. Guo, R. Gabriel, A. Nagar, R. Prasad, M. Cheng, B. Hedayatnia, A. Metallinou, et al., On evaluating and comparing open domain dialog systems (2018). arXiv:1801.03625
O. Vinyals, Q. Le, A neural conversational model (2015). arXiv:1506.05869
M. Walker, D. Litman, C.A. Kamm, A. Abella, Paradise: a framework for evaluating spoken dialogue agents, in 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics (1997), pp. 271–280
M. Walker, R.J. Passonneau, J.E. Boland, Quantitative and qualitative evaluation of darpa communicator spoken dialogue systems, in Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (2001), pp. 515–522
L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al., Ethical and social risks of harm from language models (2021). arXiv:2112.04359
B.H. Wixom, P.A. Todd, A theoretical integration of user satisfaction and technology acceptance. Inf. Syst. Res. 16(1), 85–102 (2005)
Y. Wu, W. Wu, C. Xing, M. Zhou, Z. Li, Sequential matching network: a new architecture for multi-turn response selection in retrieval-based chatbots, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2017), pp. 496–505
Y. Wu, W. Wu, D. Yang, C. Xu, Z. Li, Neural response generation with dynamic vocabularies, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)
C. Xing, W. Wu, Y. Wu, J. Liu, Y. Huang, M. Zhou, W.-Y. Ma, Topic aware neural response generation, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)
J. Xu, A. Szlam, J. Weston, Beyond goldfish memory: long-term open-domain conversation, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2022a), pp. 5180–5197
J. Xu, M. Ung, M. Komeili, K. Arora, Y.-L. Boureau, J. Weston, Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback (2022b). arXiv:2208.03270
K. Yao, G. Zweig, B. Peng, Attention with intention for a neural network conversation model (2015). arXiv:1510.08565
Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, W.B. Dolan, Dialogpt: large-scale generative pre-training for conversational response generation, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020b), pp. 270–278
W.X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models (2023). arXiv:2303.18223
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
Copyright information
© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this chapter
Cite this chapter
Mazumder, S., Liu, B. (2024). Continual Learning in Chit-Chat Systems. In: Lifelong and Continual Learning Dialogue Systems. Synthesis Lectures on Human Language Technologies. Springer, Cham. https://doi.org/10.1007/978-3-031-48189-5_5
Download citation
DOI: https://doi.org/10.1007/978-3-031-48189-5_5
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-48188-8
Online ISBN: 978-3-031-48189-5
eBook Packages: Synthesis Collection of Technology (R0)