research-article

Public Access

Open-Retrieval Conversational Question Answering

Authors:

W. Bruce Croft,

Mohit IyyerAuthors Info & Claims

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 539 - 548

https://doi.org/10.1145/3397271.3401110

Published: 25 July 2020 Publication History

Abstract

Conversational search is one of the ultimate goals of information retrieval. Recent research approaches conversational search by simplified settings of response ranking and conversational question answering, where an answer is either selected from a given candidate set or extracted from a given passage. These simplifications neglect the fundamental role of retrieval in conversational search. To address this limitation, we introduce an open-retrieval conversational question answering (ORConvQA) setting, where we learn to retrieve evidence from a large collection before extracting answers, as a further step towards building functional conversational search systems. We create a dataset, OR-QuAC, to facilitate research on ORConvQA. We build an end-to-end system for ORConvQA, featuring a retriever, a reranker, and a reader that are all based on Transformers. Our extensive experiments on OR-QuAC demonstrate that a learnable retriever is crucial for ORConvQA. We further show that our system can make a substantial improvement when we enable history modeling in all system components. Moreover, we show that the reranker component contributes to the model performance by providing a regularization effect. Finally, further in-depth analyses are performed to provide new insights into ORConvQA.

Supplementary Material

MP4 File (3397271.3401110.mp4)

Presentation video for "Open-Retrieval Conversational Question Answering"

Download
17.67 MB

References

[1]

A. Ahmad, N. Constant, Y. Yang, and D. M. Cer. ReQA: An Evaluation for End-to-End Answer Retrieval Models. ArXiv, 2019.

[2]

N. J. Belkin, C. Cool, A. Stein, and U. Thiel. Cases, Scripts, and Information-seeking Strategies: On the Design of Interactive Information Retrieval Systems. 1995.

[3]

K. Bi, Q. Ai, Y. Zhang, and W. B. Croft. Conversational Product Search Based on Negative Feedback. In CIKM, 2019.

Digital Library

[4]

D. Chen, A. Fisch, J. Weston, and A. Bordes. Reading Wikipedia to Answer Open-Domain Questions. In ACL, 2017.

[5]

Y. Chen, L. Wu, and M. J. Zaki. GraphFlow: Exploiting Conversation Flow with Graph Neural Networks for Conversational Machine Comprehension. ArXiv, 2019.

[6]

E. Choi, H. He, M. Iyyer, M. Yatskar, W.-T. Yih, Y. Choi, P. Liang, and L. Zettlemoyer. QuAC: Question Answering in Context. In EMNLP, 2018.

[7]

A. Chuklin, A. Severyn, J. R. Trippas, E. Alfonseca, H. Silén, and D. Spina. Prosody Modifications for Question-Answering in Voice-Only Settings. ArXiv, 2018.

[8]

C. Clark and M. Gardner. Simple and Effective Multi-Paragraph Reading Comprehension. In ACL, 2017.

[9]

D. Cohen, L. Yang, and W. B. Croft. WikiPassageQA: A Benchmark Collection for Research on Non-factoid Answer Passage Retrieval. In SIGIR, 2018.

Digital Library

[10]

W. B. Croft and R. H. Thompson. I3R: A New Approach to the Design of Document Retrieval Systems. JASIS, 38: 389--404, 1987.

Digital Library

[11]

R. Das, S. Dhuliawala, M. Zaheer, and A. McCallum. Multi-step Retriever-Reader Interaction for Scalable Open-domain Question Answering. In ICLR, 2019.

[12]

J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In NAACL-HLT, 2019.

[13]

B. Dhingra, K. Mazaitis, and W. W. Cohen. Quasar: Datasets for Question Answering by Search and Reading. ArXiv, 2017.

[14]

y, Cirik, and Cho]searchqaM. Dunn, L. Sagun, M. Higgins, V. U. Güney, V. Cirik, and K. Cho. SearchQA: A New Q&A Dataset Augmented with Context from a Search Engine. ArXiv, 2017.

[15]

A. Elgohary, D. Peskov, and J. L. Boyd-Graber. Can You Unpack That? Learning to Rewrite Questions-in-Context. In EMNLP/IJCNLP, 2019.

[16]

S. Garg, T. Vu, and A. Moschitti. TANDA: Transfer and Adapt Pre-Trained Transformer Models for Answer Sentence Selection. In AAAI, 2020.

[17]

P. M. Htut, S. R. Bowman, and K. Cho. Training a Ranking Function for Open-Domain Question Answering. In NAACL-HLT, 2018.

[18]

H.-Y. Huang, E. Choi, and W. tau Yih. Flowqa: Grasping flow in history for conversational machine comprehension. ArXiv, 2018.

[19]

u]faissJ. Johnson, M. Douze, and H. Jégou. Billion-scale similarity search with GPUs. ArXiv, 2017.

[20]

M. Joshi, E. Choi, D. S. Weld, and L. Zettlemoyer. TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension. In ACL, 2017.

[21]

V. Karpukhin, B. Ouguz, S. Min, L. Y. Wu, S. Edunov, D. Chen, and W. tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. ArXiv, abs/2004.04906, 2020.

[22]

B. Kratzwald and S. Feuerriegel. Adaptive Document Retrieval for Deep Question Answering. In EMNLP, 2018.

[23]

T. Kwiatkowski, J. Palomaki, O. Redfield, M. Collins, A. P. Parikh, C. Alberti, D. Epstein, I. Polosukhin, J. Devlin, K. Lee, K. Toutanova, L. Jones, M. Kelcey, M.-W. Chang, A. M. Dai, J. Uszkoreit, Q. Le, and S. Petrov. Natural Questions: A Benchmark for Question Answering Research. TACL, 7: 453--466, 2019.

[24]

Z.-Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. ArXiv, 2019.

[25]

J. Lee, S. Yun, H. Kim, M. Ko, and J. Kang. Ranking Paragraphs for Improving Answer Recall in Open-Domain Question Answering. In EMNLP, 2018.

[26]

K. Lee, M.-W. Chang, and K. Toutanova. Latent Retrieval for Weakly Supervised Open Domain Question Answering. In ACL, 2019.

[27]

R. Lowe, N. Pow, I. Serban, and J. Pineau. The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systems. In SIGDIAL, 2015.

[28]

T. Nguyen, M. Rosenberg, X. Song, J. Gao, S. Tiwary, R. Majumder, and L. Deng. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. ArXiv, 2016.

[29]

R. N. Oddy. Information Retrieval through Man-Machine Dialogue. 1977.

[30]

C. Qu, L. Yang, W. B. Croft, J. R. Trippas, Y. Zhang, and M. Qiu. Analyzing and Characterizing User Intent in Information-seeking Conversations. In SIGIR, 2018.

Digital Library

[31]

Qu, Yang, Croft, Scholer, and Zhang]answer_interactionC. Qu, L. Yang, W. B. Croft, F. Scholer, and Y. Zhang. Answer Interaction in Non-factoid Question Answering Systems. In CHIIR, 2019 a.

Digital Library

[32]

Qu, Yang, Croft, Zhang, Trippas, and Qiu]UserIntentPredC. Qu, L. Yang, W. B. Croft, Y. Zhang, J. R. Trippas, and M. Qiu. User Intent Prediction in Information-seeking Conversations. In CHIIR, 2019 b.

Digital Library

[33]

Qu, Yang, Qiu, Croft, Zhang, and Iyyer]haeC. Qu, L. Yang, M. Qiu, W. B. Croft, Y. Zhang, and M. Iyyer. BERT with History Answer Embedding for Conversational Question Answering. In SIGIR, 2019 c.

Digital Library

[34]

Qu, Yang, Qiu, Zhang, Chen, Croft, and Iyyer]hamC. Qu, L. Yang, M. Qiu, Y. Zhang, C. Chen, W. B. Croft, and M. Iyyer. Attentive History Selection for Conversational Question Answering. In CIKM, 2019 d.

Digital Library

[35]

P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang. SQuAD: 100,000 Questions for Machine Comprehension of Text. In EMNLP, 2016.

[36]

P. Rajpurkar, R. Jia, and P. Liang. Know What You Don't Know: Unanswerable Questions for SQuAD. In ACL, 2018.

[37]

S. Reddy, D. Chen, and C. D. Manning. CoQA: A Conversational Question Answering Challenge. TACL, 7: 249--266, 2018.

[38]

A. Shrivastava and P. Li. Asymmetric LSH (ALSH) for Sublinear Time Maximum Inner Product Search (MIPS). In NIPS, 2014.

Digital Library

[39]

C. Tao, W. Wu, C. Xu, W. Hu, D. Zhao, and R. Yan. Multi-Representation Fusion Network for Multi-Turn Response Selection in Retrieval-Based Chatbots. In WSDM, 2019.

Digital Library

[40]

P. Thomas, D. J. McDuff, M. Czerwinski, and N. Craswell. MISC: A data set of information-seeking conversations. In SIGIR (CAIR'17), 2017.

[41]

J. R. Trippas, D. Spina, L. Cavedon, and M. Sanderson. How Do People Interact in Conversational Speech-Only Search Tasks: A Preliminary Analysis. In CHIIR, 2017.

Digital Library

[42]

J. R. Trippas, D. Spina, L. Cavedon, H. Joho, and M. Sanderson. Informing the Design of Spoken Conversational Search: Perspective Paper. In CHIIR, 2018.

Digital Library

[43]

J. R. Trippas, D. Spina, P. Thomas, M. Sanderson, H. Joho, and L. Cavedon. Towards a Model for Spoken Conversational Search. ArXiv, 2019.

[44]

A. Trischler, T. Wang, X. Yuan, J. Harris, A. Sordoni, P. Bachman, and K. Suleman. NewsQA: A Machine Comprehension Dataset. In Rep4NLP@ACL, 2016.

[45]

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention Is All You Need. In NIPS, 2017.

Digital Library

[46]

E. M. Voorhees and D. M. Tice. The TREC-8 Question Answering Track Evaluation. In TREC, 1999.

[47]

M. Wang, N. A. Smith, and T. Mitamura. What is the Jeopardy Model? A Quasi-Synchronous Grammar for QA. In EMNLP-CoNLL, 2007.

[48]

S. Wang, M. Yu, X. Guo, Z. Wang, T. Klinger, W. Zhang, S. Chang, G. Tesauro, B. Zhou, and J. Jiang. R3: Reinforced Ranker-Reader for Open-Domain Question Answering. In AAAI, 2018.

[49]

Y. Wu, W. Y. Wu, M. Zhou, and Z. Li. Sequential Match Network: A New Architecture for Multi-turn Response Selection in Retrieval-based Chatbots. In ACL, 2016.

[50]

Yan, Song, and Wu]Yan2016LearningTRR. Yan, Y. Song, and H. Wu. Learning to Respond with Deep Neural Networks for Retrieval-Based Human-Computer Conversation System. In SIGIR, 2016.

Digital Library

[51]

Yan, Song, Zhou, and Wu]Yan2016ShallIBR. Yan, Y. Song, X. Zhou, and H. Wu. "Shall I Be Your Chat Companion?": Towards an Online Human-Computer Conversation System. In CIKM, 2016 b.

Digital Library

[52]

L. Yang, H. Zamani, Y. Zhang, J. Guo, and W. B. Croft. Neural Matching Models for Question Retrieval and Next Question Prediction in Conversation. ArXiv, 2017.

[53]

L. Yang, M. Qiu, C. Qu, J. Guo, Y. Zhang, W. B. Croft, J. Huang, and H. Chen. Response Ranking with Deep Matching Networks and External Knowledge in Information-seeking Conversation Systems. In SIGIR, 2018.

Digital Library

[54]

Yang, Hu, Qiu, Qu, Gao, Croft, Liu, Shen, and Liu]hybridL. Yang, J. Hu, M. Qiu, C. Qu, J. Gao, W. B. Croft, X. Liu, Y. Shen, and J. Liu. A Hybrid Retrieval-Generation Neural Conversation Model. In CIKM, 2019 a.

Digital Library

[55]

L. Yang, M. Qiu, C. Qu, C. Chen, J. Guo, Y. Zhang, W. B. Croft, and H. Chen. IART: Intent-aware Response Ranking with Transformers in Information-seeking Conversation Systems. In WWW, 2020.

Digital Library

[56]

Yang, Xie, Lin, Li, Tan, Xiong, Li, and Lin]bertseriniW. Yang, Y. Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, and J. Lin. End-to-End Open-Domain Question Answering with BERTserini. In NAACL-HLT, 2019 b.

[57]

Y. Yang, W.-T. Yih, and C. Meek. WikiQA: A Challenge Dataset for Open-Domain Question Answering. In EMNLP, 2015.

[58]

M. Yatskar. A Qualitative Comparison of CoQA, SQuAD 2.0 and QuAC. In NAACL-HLT, 2018.

[59]

Y.-T. Yeh and Y.-N. Chen. FlowDelta: Modeling Flow Information Gain in Reasoning for Conversational Machine Comprehension. ArXiv, 2019.

[60]

Y. Zhang, X. Chen, Q. Ai, L. Yang, and W. B. Croft. Towards Conversational Search and Recommendation: System Ask, User Respond. In CIKM, 2018.

[61]

C. Zhu, M. Zeng, and X. Huang. SDNet: Contextualized Attention-based Deep Network for Conversational Question Answering. ArXiv, 2018.

Cited By

Samarinas CZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ProCIS: A Benchmark for Proactive Retrieval in ConversationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657869(830-840)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657869
Abbasiantaeb ZYuan YKanoulas EAliannejadi MAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM InteractionsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635856(8-17)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635856
Mo FYi BMao KQu CHuang KNie JChua TNgo CKumar RLauw HKa-Wei Lee R(2024)ConvSDG: Session Data Generation for Conversational SearchCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651940(1634-1642)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651940
Show More Cited By

Index Terms

Open-Retrieval Conversational Question Answering
1. Computing methodologies
  1. Artificial intelligence
    1. Natural language processing
      1. Discourse, dialogue and pragmatics
2. Information systems
  1. Information retrieval
    1. Retrieval tasks and goals
      1. Question answering

Recommendations

Question Rewriting for Conversational Question Answering
WSDM '21: Proceedings of the 14th ACM International Conference on Web Search and Data Mining

Conversational question answering (QA) requires the ability to correctly interpret a question in the context of previous conversation turns. We address the conversational QA task by decomposing it into question rewriting and question answering subtasks. ...
BERT with History Answer Embedding for Conversational Question Answering
SIGIR'19: Proceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

Conversational search is an emerging topic in the information retrieval community. One of the major challenges to multi-turn conversational search is to model the conversation history to answer the current question. Existing methods either prepend ...
Attentive History Selection for Conversational Question Answering
CIKM '19: Proceedings of the 28th ACM International Conference on Information and Knowledge Management

Conversational question answering (ConvQA) is a simplified but concrete setting of conversational search. One of its major challenges is to leverage the conversation history to understand and answer the current question. In this work, we propose a novel ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2020

2548 pages

ISBN:9781450380164

DOI:10.1145/3397271

General Chairs:
Jimmy Huang
York University, Canada
,
Yi Chang
Jilin University, China
,
Xueqi Cheng
Chinese Academy of Sciences, China
,
Program Chairs:
Jaap Kamps
University of Amsterdam, Netherlands
,
Vanessa Murdock
Amazon, U.S.A.
,
Ji-Rong Wen
Renmin University of China, China
,
Yiqun Liu
Tsinghua University, China

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 July 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SIGIR '20

Sponsor:

SIGIR

SIGIR '20: The 43rd International ACM SIGIR conference on research and development in Information Retrieval

July 25 - 30, 2020

Virtual Event, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

48
Total Citations
View Citations
2,183
Total Downloads

Downloads (Last 12 months)547
Downloads (Last 6 weeks)39

Reflects downloads up to 11 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Samarinas CZamani HHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)ProCIS: A Benchmark for Proactive Retrieval in ConversationsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657869(830-840)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657869
Abbasiantaeb ZYuan YKanoulas EAliannejadi MAngélica LLattanzi SMuñoz Medina AAkoglu LGionis AVassilvitskii S(2024)Let the LLMs Talk: Simulating Human-to-Human Conversational QA via Zero-Shot LLM-to-LLM InteractionsProceedings of the 17th ACM International Conference on Web Search and Data Mining10.1145/3616855.3635856(8-17)Online publication date: 4-Mar-2024
https://dl.acm.org/doi/10.1145/3616855.3635856
Mo FYi BMao KQu CHuang KNie JChua TNgo CKumar RLauw HKa-Wei Lee R(2024)ConvSDG: Session Data Generation for Conversational SearchCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3651940(1634-1642)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3651940
Zamiri MQiang YNikolaev FZhu DKotov AChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Benchmark and Neural Architecture for Conversational Entity Retrieval from a Knowledge GraphProceedings of the ACM on Web Conference 202410.1145/3589334.3645676(1519-1528)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645676
Liu ZHe JGong TWeng HWang FLiu HHao T(2024)Improving Topic Tracing with a Textual Reader for Conversational Knowledge Based Question AnsweringIEEE Transactions on Emerging Topics in Computational Intelligence10.1109/TETCI.2024.33694788:3(2640-2653)Online publication date: Jun-2024
https://doi.org/10.1109/TETCI.2024.3369478
Rashid MMeem JHristidis V(2024)NORMY: Non-Uniform History Modeling for Open Retrieval Conversational Question Answering2024 IEEE 18th International Conference on Semantic Computing (ICSC)10.1109/ICSC59802.2024.00022(101-109)Online publication date: 5-Feb-2024
https://doi.org/10.1109/ICSC59802.2024.00022
Wan HLi HLu SCui XDanilevsky M(2024)How Can Personalized Context Help? Exploring Joint Retrieval of Passage and Personalized ContextICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP48485.2024.10447921(9991-9995)Online publication date: 14-Apr-2024
https://doi.org/10.1109/ICASSP48485.2024.10447921
Andreasen TBordogna GTré GKacprzyk JLarsen HZadrożny S(2024)The power and potentials of Flexible Query Answering Systems: A critical and comprehensive analysisData & Knowledge Engineering10.1016/j.datak.2023.102246149(102246)Online publication date: Jan-2024
https://doi.org/10.1016/j.datak.2023.102246
Arabzadeh NBigdeli ABagheri E(2024)LaQuE: Enabling Entity Search at ScaleAdvances in Information Retrieval10.1007/978-3-031-56060-6_18(270-285)Online publication date: 16-Mar-2024
https://doi.org/10.1007/978-3-031-56060-6_18
Wang XSen PLi RYilmaz E(2024)Simulated Task Oriented Dialogues for Developing Versatile Conversational AgentsAdvances in Information Retrieval10.1007/978-3-031-56027-9_10(157-172)Online publication date: 20-Mar-2024
https://doi.org/10.1007/978-3-031-56027-9_10
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents