Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Continual Learning in Chit-Chat Systems

  • Chapter
  • First Online:
Lifelong and Continual Learning Dialogue Systems

Part of the book series: Synthesis Lectures on Human Language Technologies ((SLHLT))

  • 85 Accesses

Abstract

Chit-chat dialogue systems, also known as open-domain dialogue systems, focus on carrying out chit-chat type of conversations with users on any topic without specific goals to complete (see Sect. 1.1 for more details). The need to support such free-flow conversations often makes it challenging to build dialogue systems that can perform well in practice. Existing methods for building chit-chat systems mainly rely on collecting a dialogue corpus with context and response pairs and training a response generation model. However, no matter how big a corpus is collected for the training, it will always be limited in terms of the knowledge that the chatbot needs to successfully model relevant responses in post-deployment real-world conversations with users. The ability to continually learn from post-deployment conversation experiences and self-improve the response generation performance thus become essential for the success of commercial chit-chat systems. This chapter focuses on the topic.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 34.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Hardcover Book
USD 44.99
Price excludes VAT (USA)
  • Durable hardcover edition
  • Dispatched in 3 to 5 business days
  • Free shipping worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Notes

  1. 1.

    https://openai.com/blog/chatgpt

    https://chat.openai.com/chat.

  2. 2.

    https://openai.com/research/gpt-4.

References

  • R. Aljundi, L. Caccia, E. Belilovsky, M. Caccia, M. Lin, L. Charlin, T. Tuytelaars, Online continual learning with maximally interfered retrieval, in Advances in Neural Information Processing Systems (2019)

    Google Scholar 

  • S. Bao, H. He, F. Wang, W. Hua, H. Wang, W. Wenquan, W. Zhihua, Z. Guo, L. Hua, X. Huang, et al., Plato-xl: Exploring the large-scale pre-training of dialogue generation, in Findings of the Association for Computational Linguistics: AACL-IJCNLP (2022), pp. 107–118

    Google Scholar 

  • E.M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the dangers of stochastic parrots: can language models be too big? in Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency (2021), pp. 610–623

    Google Scholar 

  • R. Bommasani, D.A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M.S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, et al., On the opportunities and risks of foundation models (2021). arXiv:2108.07258

  • Braden Hancock, Antoine Bordes, Pierre-Emmanuel Mazare, and Jason Weston. Learning from dialogue after deployment: Feed yourself, chatbot! In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics (2019), pp. 3667–3684

    Google Scholar 

  • T. Brown, B. Mann, N. Ryder, M. Subbiah, J.D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., Language models are few-shot learners, in Advances in Neural Information Processing Systems 33 (2020), pp. 1877–1901

    Google Scholar 

  • A. Chaudhry, M. Rohrbach, M. Elhoseiny, T. Ajanthan, P.K. Dokania, P.H.S. Torr, M. Ranzato, Continual learning with tiny episodic memories, in Workshop on Multi-Task and Lifelong Reinforcement Learning (2019)

    Google Scholar 

  • H. Chen, X. Liu, D. Yin, J. Tang, A survey on dialogue systems: recent advances and new frontiers. ACM SIGKDD Explorations Newsl 19(2), 25–35 (2017)

    Article  Google Scholar 

  • G. Chen, X. Li, S. Xiao, C. Zhang, L. Xianghua, Racl: a robust adaptive contrastive learning method for conversational satisfaction prediction. Pattern Recognit. 138, 109386 (2023)

    Article  Google Scholar 

  • J.I. Choi, E. Agichtein, Quantifying the effects of prosody modulation on user engagement and satisfaction in conversational systems, in Proceedings of the 2020 Conference on Human Information Interaction and Retrieval (2020), pp. 417–421

    Google Scholar 

  • J.I. Choi, A. Ahmadvand, E. Agichtein, Offline and online satisfaction prediction in open-domain conversational systems, in Proceedings of the 28th ACM International Conference on Information and Knowledge Management (2019), pp. 1281–1290

    Google Scholar 

  • P.F. Christiano, J. Leike, T. Brown, M. Martic, S. Legg, D. Amodei, Deep reinforcement learning from human preferences, in Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

  • W.H. DeLone, E.R. McLean, Information systems success: the quest for the dependent variable. Inf. Syst. Res. 3(1), 60–95 (1992)

    Article  Google Scholar 

  • Y. Deng, W. Zhang, W. Lam, H. Cheng, H. Meng, User satisfaction estimation with sequential dialogue act modeling in goal-oriented conversational systems, in Proceedings of the ACM Web Conference (2022), pp. 2998–3008

    Google Scholar 

  • R. Gabriel, Y. Liu, A. Gottardi, M. Eric, A. Khatri, A. Chadha, Q. Chen, B. Hedayatnia, P. Rajan, A. Binici, et al., Further advances in open domain dialog systems in the third alexa prize socialbot grand challenge (2019)

    Google Scholar 

  • S. Gehman, S. Gururangan, M. Sap, Y. Choi, N.A. Smith, Realtoxicityprompts: evaluating neural toxic degeneration in language models, in Findings of the Association for Computational Linguistics: EMNLP (2020), pp. 3356–3369

    Google Scholar 

  • Y. Guo, B. Liu, D. Zhao, Online continual learning through mutual information maximization, in International Conference on Machine Learning (PMLR, 2022), pp. 8109–8126

    Google Scholar 

  • S.H. Hashemi, K. Williams, A. El Kholy, I. Zitouni, P.A. Crook, Measuring user satisfaction on smart speaker intelligent assistants using intent sensitive query embeddings, in Proceedings of the 27th ACM International Conference on Information and Knowledge Management (2018), pp. 1183–1192

    Google Scholar 

  • S. Humeau, K. Shuster, M.-A. Lachaux, J. Weston, Poly-encoders: transformer architectures and pre-training strategies for fast and accurate multi-sentence scoring, in International Conference on Learning Representations (2019)

    Google Scholar 

  • J. Johnson, M. Douze, H. Jégou, Billion-scale similarity search with gpus. IEEE Trans. Big Data 7(3), 535–547 (2019)

    Article  Google Scholar 

  • D. Ju, J. Xu, Y.-L. Boureau, J. Weston, Learning from data in the mixed adversarial non-adversarial case: finding the helpers and ignoring the trolls (2022). arXiv:2208.03295

  • Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, G. Irving, Alignment of language agents (2021). arXiv:2103.14659

  • J. Kiseleva, K. Williams, J. Jiang, A.H. Awadallah, A.C. Crook, I. Zitouni, T. Anastasakos, Understanding user satisfaction with intelligent assistants, in Proceedings of the 2016 ACM on Conference on Human Information Interaction and Retrieval (2016), pp. 121–130

    Google Scholar 

  • M. Komeili, K. Shuster, J. Weston, Internet-augmented dialogue generation, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2022), pp. 8460–8478

    Google Scholar 

  • J. Leike, D. Krueger, T. Everitt, M. Martic, V. Maini, S. Legg, Scalable agent alignment via reward modeling: a research direction (2018). arXiv:1811.07871

  • J. Li, M. Galley, C. Brockett, J. Gao, W.B. Dolan, A diversity-promoting objective function for neural conversation models, in Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (2016a), pp. 110–119

    Google Scholar 

  • J. Li, W. Monroe, A. Ritter, D. Jurafsky, M. Galley, J. Gao, Deep reinforcement learning for dialogue generation, in Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing (2016b), pp. 1192–1202

    Google Scholar 

  • R. Liang, R. Takanobu, F.-L. Li, J. Zhang, H. Chen, M. Huang, Turn-level user satisfaction estimation in e-commerce customer service, in Proceedings of the 4th Workshop on e-Commerce and NLP (2021), pp. 26–32

    Google Scholar 

  • A. Lipani, B. Carterette, E. Yilmaz, How am i doing?: Evaluating conversational search systems offline. ACM Trans. Inf. Syst. (TOIS) 39(4), 1–22 (2021)

    Article  Google Scholar 

  • J. Lopes, How generic can dialogue breakdown detection be? the kth entry to dbdc3, in Proceedings of Dialog System Technology Challenge 6 (2017)

    Google Scholar 

  • H. Lu, S. Bao, H. He, F. Wang, H. Wu, H. Wang, Towards boosting the open-domain chatbot with human feedback (2022). arXiv:2208.14165

  • H. Mei, M. Bansal, M. Walter, Coherent dialogue with attention-based language models, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  • OpenAI, Gpt-4 technical report (2023). arXiv:2303.08774 [cs.CL]

  • L. Ouyang, W. Jeffrey, X. Jiang, D. Almeida, C. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray et al., Training language models to follow instructions with human feedback, in Advances in Neural Information Processing Systems 35 (2022), pp. 27730–27744

    Google Scholar 

  • G. Pandey, D. Contractor, V. Kumar, S. Joshi, Exemplar encoder-decoder for neural conversation generation, in Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2018), pp. 1329–1338

    Google Scholar 

  • A. Paranjape, A. See, K. Kenealy, H. Li, A. Hardy, P. Qi, K.R. Sadagopan, N.M. Phu, D. Soylu, C.D. Manning, Neural generation meets real people: towards emotionally engaging mixed-initiative conversations, in 3rd Proceedings of Alexa Prize (Alexa Prize 2019) (2020)

    Google Scholar 

  • A. Ram, R. Prasad, C. Khatri, A. Venkatesh, R. Gabriel, Q. Liu, J. Nunn, B. Hedayatnia, M. Cheng, A. Nagar, et al., Conversational ai: the science behind the alexa prize (2018)

    Google Scholar 

  • S. Roller, E. Dinan, N. Goyal, D. Ju, M. Williamson, Y. Liu, J. Xu, M. Ott, E.M. Smith, Y.-L. Boureau, et al., Recipes for building an open-domain chatbot, in Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume (2021), pp. 300–325

    Google Scholar 

  • H. Ryuichiro, K. Funakoshi, M. Inaba, Y. Tsunomori, T. Takahashi, K. Nobuhiro, Overview of dialogue breakdown detection challenge 3, in Proceedings of Dialogue System Technology Challenge (2017), p. 14

    Google Scholar 

  • T. Sandbank, M. Shmueli-Scheuer, J. Herzig, D. Konopnicki, J. Richards, D. Piorkowski, Detecting egregious conversations between customers and virtual agents, in Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers) (2018), pp. 1802–1811

    Google Scholar 

  • J. Schulman, F. Wolski, P. Dhariwal, A. Radford, O. Klimov, Proximal policy optimization algorithms (2017). arXiv:1707.06347

  • A. See, C.D. Manning, Understanding and predicting user dissatisfaction in a neural generative chatbot, in Proceedings of the 22nd Annual Meeting of the Special Interest Group on Discourse and Dialogue (2021), pp. 1–12

    Google Scholar 

  • I.V. Serban, A. Sordoni, Y. Bengio, A. Courville, J. Pineau, Building end-to-end dialogue systems using generative hierarchical neural network models, in Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence (2016), pp. 3776–3783

    Google Scholar 

  • I.V. Serban, R. Lowe, P. Henderson, L. Charlin, J. Pineau, A survey of available corpora for building data-driven dialogue systems: The journal version. Dialogue & Discourse 9(1), 1–49 (2018)

    Article  Google Scholar 

  • L. Shang, Z. Lu, H. Li, Neural responding machine for short-text conversation, in Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (2015), pp. 1577–1586

    Google Scholar 

  • X. Shen, H. Su, Y. Li, W. Li, S. Niu, Y. Zhao, A. Aizawa, G. Long, A conditional variational framework for dialog generation, in 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017 (Association for Computational Linguistics (ACL), 2017), pp. 504–509

    Google Scholar 

  • X. Shen, H. Su, S. Niu, V. Demberg, Improving variational encoder-decoders in dialogue generation, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  • W. Shi, E. Dinan, K. Shuster, J. Weston, J. Xu, When life gives you lemons, make cherryade: converting feedback from bad responses into good labels (2022). arXiv:2210.15893

  • K. Shuster, M. Komeili, L. Adolphs, S. Roller, A. Szlam, J. Weston, Language models that seek for knowledge: modular search & generation for dialogue and prompt completion, in Findings of the Association for Computational Linguistics: EMNLP 2022 (2022)

    Google Scholar 

  • K. Shuster, J. Urbanek, E. Dinan, A. Szlam, J. Weston, Dialogue in the wild: learning from a deployed role-playing game with humans and bots, in Findings of the Association for Computational Linguistics: ACL-IJCNLP (2021), pp. 611–624

    Google Scholar 

  • K. Shuster, J. Xu, M. Komeili, D. Ju, E.M. Smith, S. Roller, M. Ung, M. Chen, K. Arora, J. Lane, et al., Blenderbot 3: a deployed conversational agent that continually learns to responsibly engage (2022). arXiv:2208.03188

  • S. Steidl, C. Hacker, C. Ruff, A. Batliner, E. Nöth, J. Haas, Looking at the last two turns, i’d say this dialogue is doomed–measuring dialogue success, in Text, Speech and Dialogue: 7th International Conference, TSD 2004, Brno, Czech Republic, September 8–11, 2004. Proceedings 7 (Springer, 2004), pp. 629–636

    Google Scholar 

  • N. Stiennon, L. Ouyang, J. Wu, D. Ziegler, R. Lowe, C. Voss, A. Radford, D. Amodei, P.F. Christiano, Learning to summarize with human feedback, in Advances in Neural Information Processing Systems 33 (2020), pp. 3008–3021

    Google Scholar 

  • Y. Sun, L. Wu, S. Song, X. Yu, X. He, G. Fu, Tracking satisfaction states for customer satisfaction prediction in e-commerce service chatbots, in Proceedings of the 29th International Conference on Computational Linguistics (2022), pp. 616–625

    Google Scholar 

  • A. Tamkin, M. Brundage, J. Clark, D. Ganguli, Understanding the capabilities, limitations, and societal impact of large language models (2021). arXiv:2102.02503

  • J. Urbanek, A. Fan, S. Karamcheti, S. Jain, S. Humeau, E. Dinan, T. Rocktäschel, D. Kiela, A. Szlam, J. Weston, Learning to speak and act in a fantasy text adventure game, in Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP) (2010) (2019), pp. 673–683

    Google Scholar 

  • A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A.N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in Advances in Neural Information Processing Systems 30 (2017)

    Google Scholar 

  • A. Venkatesh, C. Khatri, A. Ram, F. Guo, R. Gabriel, A. Nagar, R. Prasad, M. Cheng, B. Hedayatnia, A. Metallinou, et al., On evaluating and comparing open domain dialog systems (2018). arXiv:1801.03625

  • O. Vinyals, Q. Le, A neural conversational model (2015). arXiv:1506.05869

  • M. Walker, D. Litman, C.A. Kamm, A. Abella, Paradise: a framework for evaluating spoken dialogue agents, in 35th Annual Meeting of the Association for Computational Linguistics and 8th Conference of the European Chapter of the Association for Computational Linguistics (1997), pp. 271–280

    Google Scholar 

  • M. Walker, R.J. Passonneau, J.E. Boland, Quantitative and qualitative evaluation of darpa communicator spoken dialogue systems, in Proceedings of the 39th Annual Meeting of the Association for Computational Linguistics (2001), pp. 515–522

    Google Scholar 

  • L. Weidinger, J. Mellor, M. Rauh, C. Griffin, J. Uesato, P.-S. Huang, M. Cheng, M. Glaese, B. Balle, A. Kasirzadeh, et al., Ethical and social risks of harm from language models (2021). arXiv:2112.04359

  • B.H. Wixom, P.A. Todd, A theoretical integration of user satisfaction and technology acceptance. Inf. Syst. Res. 16(1), 85–102 (2005)

    Article  Google Scholar 

  • Y. Wu, W. Wu, C. Xing, M. Zhou, Z. Li, Sequential matching network: a new architecture for multi-turn response selection in retrieval-based chatbots, in Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2017), pp. 496–505

    Google Scholar 

  • Y. Wu, W. Wu, D. Yang, C. Xu, Z. Li, Neural response generation with dynamic vocabularies, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32 (2018)

    Google Scholar 

  • C. Xing, W. Wu, Y. Wu, J. Liu, Y. Huang, M. Zhou, W.-Y. Ma, Topic aware neural response generation, in Proceedings of the AAAI Conference on Artificial Intelligence, vol. 31 (2017)

    Google Scholar 

  • J. Xu, A. Szlam, J. Weston, Beyond goldfish memory: long-term open-domain conversation, in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (2022a), pp. 5180–5197

    Google Scholar 

  • J. Xu, M. Ung, M. Komeili, K. Arora, Y.-L. Boureau, J. Weston, Learning new skills after deployment: Improving open-domain internet-driven dialogue with human feedback (2022b). arXiv:2208.03270

  • K. Yao, G. Zweig, B. Peng, Attention with intention for a neural network conversation model (2015). arXiv:1510.08565

  • Y. Zhang, S. Sun, M. Galley, Y.-C. Chen, C. Brockett, X. Gao, J. Gao, J. Liu, W.B. Dolan, Dialogpt: large-scale generative pre-training for conversational response generation, in Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations (2020b), pp. 270–278

    Google Scholar 

  • W.X. Zhao, K. Zhou, J. Li, T. Tang, X. Wang, Y. Hou, Y. Min, B. Zhang, J. Zhang, Z. Dong, et al., A survey of large language models (2023). arXiv:2303.18223

Download references

Author information

Authors and Affiliations

  1. Intel Labs, Santa Clara, CA, USA

    Sahisnu Mazumder

  2. University of Illinois Chicago, Chicago, IL, USA

    Bing Liu

Authors
  1. Sahisnu Mazumder
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Bing Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Sahisnu Mazumder .

Rights and permissions

Reprints and permissions

Copyright information

© 2024 The Author(s), under exclusive license to Springer Nature Switzerland AG

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Mazumder, S., Liu, B. (2024). Continual Learning in Chit-Chat Systems. In: Lifelong and Continual Learning Dialogue Systems. Synthesis Lectures on Human Language Technologies. Springer, Cham. https://doi.org/10.1007/978-3-031-48189-5_5

Download citation

  • DOI: https://doi.org/10.1007/978-3-031-48189-5_5

  • Published:

  • Publisher Name: Springer, Cham

  • Print ISBN: 978-3-031-48188-8

  • Online ISBN: 978-3-031-48189-5

  • eBook Packages: Synthesis Collection of Technology (R0)

Publish with us

Policies and ethics

Search

Navigation