research-article

Open access

Stochastic Retrieval-Conditioned Reranking

Authors:

Michael Bendersky,

Donald Metzler,

Honglei Zhuang,

Xuanhui WangAuthors Info & Claims

ICTIR '22: Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval

Pages 81 - 91

https://doi.org/10.1145/3539813.3545141

Published: 25 August 2022 Publication History

Abstract

The multi-stage cascaded architecture has been adopted by many search engines for efficient and effective retrieval. This architecture consists of a stack of retrieval and reranking models in which efficient retrieval models are followed by effective (neural) learning-to-rank models. The optimization of these learning-to-rank models is loosely connected to the early stage retrieval models. This paper draws theoretical connections between the early stage retrieval and late stage reranking models by deriving expected reranking performance conditioned on the early stage retrieval results. Our findings shed light on optimization of both retrieval and reranking models. As a result, we also introduce a novel loss function for training reranking models that leads to significant improvements on multiple public benchmarks. Our findings provide theoretical and empirical guidelines for developing multi-stage cascaded retrieval models.

References

[1]

Avi Arampatzis and André van Hameran. The score-distributional threshold optimization for adaptive binary classification tasks. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 285--293, 2001.

Digital Library

[2]

Nima Asadi (Sebastian Bruch) and Jimmy Lin. Fast candidate generation for two-phase document ranking: Postings list intersection with bloom filters. In Proceedings of the 21st ACM international conference on Information and knowledge management, pages 2419--2422, 2012.

[3]

Javed A Aslam, Evangelos Kanoulas, Virgil Pavlu, Stefan Savev, and Emine Yilmaz. Document selection methodologies for efficient and effective learning-to-rank. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, pages 468--475, 2009.

Digital Library

[4]

Dara Bahri, Yi Tay, Che Zheng, Don Metzler, and Andrew Tomkins. Choppy: Cut transformers for ranked list truncation. 2020.

[5]

Michael Bendersky, Honglei Zhuang, Ji Ma, Shuguang Han, Keith Hall, and Ryan McDonald. Rrf102: Meeting the trec-covid challenge with a 100+ runs ensemble, 2020.

[6]

Andrei Z. Broder, David Carmel, Michael Herscovici, Aya Soffer, and Jason Zien. Efficient query evaluation using a two-level retrieval process. In Proceedings of the Twelfth International Conference on Information and Knowledge Management, CIKM '03, page 426--434, New York, NY, USA, 2003. Association for Computing Machinery.

Digital Library

[7]

Sebastian Bruch, Shuguang Han, Michael Bendersky, and Marc Najork. A stochastic treatment of learning to rank scoring functions. In Proceedings of the 13th International Conference on Web Search and Data Mining, page 61--69, New York, NY, USA, 2020. Association for Computing Machinery.

Digital Library

[8]

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. Learning to rank: from pairwise approach to listwise approach. In Proceedings of the 24th international conference on Machine learning, pages 129--136, 2007.

Digital Library

[9]

Olivier Chapelle and Yi Chang. Yahoo! learning to rank challenge overview. In Proceedings of the learning to rank challenge, pages 1--24. PMLR, 2011.

[10]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2019 deep learning track. In TREC, 2019.

[11]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, and Daniel Campos. Overview of the trec 2020 deep learning track. In TREC, 2020.

[12]

Nick Craswell, Bhaskar Mitra, Emine Yilmaz, Daniel Campos, and Jimmy Lin. Ms marco: Benchmarking ranking models in the large-data regime. In Proceedings of the 44th international ACM SIGIR conference on Research & development in information retrieval. ACM, April 2021.

Digital Library

[13]

Bruce Croft, Donald Metzler, and Trevor Strohman. Search Engines: Information Retrieval in Practice. Addison-Wesley Publishing Company, USA, 1st edition, 2009.

Digital Library

[14]

Van Dang, Michael Bendersky, and W Bruce Croft. Two-stage learning to rank for information retrieval. In European Conference on Information Retrieval, pages 423--434. Springer, 2013.

Digital Library

[15]

J. Devlin, M. Chang, K. Lee, and K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, 2019.

[16]

Fernando Diaz. Regularizing ad hoc retrieval scores. In Proceedings of the 14th ACM International Conference on Information and Knowledge Management, CIKM '05, page 672--679, New York, NY, USA, 2005. Association for Computing Machinery.

Digital Library

[17]

Fernando Diaz, Bhaskar Mitra, Michael D. Ekstrand, Asia J. Biega, and Ben Carterette. Evaluating stochastic rankings with expected exposure. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, page 275--284, NewYork, NY, USA, 2020. Association for Computing Machinery.

Digital Library

[18]

Luke Gallagher, Ruey-Cheng Chen, Roi Blanco, and J. Shane Culpepper. Joint optimization of cascade ranking models. In Proceedings of the Twelfth ACM International Conference on Web Search and Data Mining, WSDM '19, page 15--23, New York, NY, USA, 2019. Association for Computing Machinery.

Digital Library

[19]

Helia Hashemi, Mohammad Aliannejadi, Hamed Zamani, and W Bruce Croft. Antique: A non-factoid question answering benchmark. In Proceedings of the 2020 European Conference on Information Retrieval, ECIR '20, 2020.

Digital Library

[20]

Sebastian Hofstätter, Sophia Althammer, Michael Schröder, Mete Sertkan, and Allan Hanbury. Improving efficient neural ranking models with cross-architecture knowledge distillation. CoRR, abs/2010.02666, 2020.

[21]

Sebastian Hofstätter, Sheng-Chieh Lin, Jheng-Hong Yang, Jimmy Lin, and Allan Hanbury. Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Sampling, page 113--122. SIGIR '21. Association for Computing Machinery, New York, NY, USA, 2021.

Digital Library

[22]

Sebastian Hofstätter, Bhaskar Mitra, Hamed Zamani, Nick Craswell, and Allan Hanbury. Intra-document cascading: Learning to select passages for neural document ranking. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 1349--1358, New York, NY, USA, 2021. Association for Computing Machinery.

[23]

Sebastian Hofstätter, Hamed Zamani, Bhaskar Mitra, Nick Craswell, and Allan Hanbury. Local Self-Attention over Long Text for Efficient Document Retrieval, page 2021--2024. Association for Computing Machinery, New York, NY, USA, 2020.

[24]

Kalervo Järvelin and Jaana Kekäläinen. Cumulated gain-based evaluation of ir techniques. ACM Trans. Inf. Syst., 20(4):422--446, 2002.

Digital Library

[25]

Evangelos Kanoulas, Keshi Dai, Virgil Pavlu, and Javed A. Aslam. Score distribution models: Assumptions, intuition, and robustness to score manipulation. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '10, page 242--249, New York, NY, USA, 2010. Association for Computing Machinery.

Digital Library

[26]

Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, andWen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769--6781, Online, November 2020. Association for Computational Linguistics.

[27]

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Porceedings of the 3rd International Conference on Learning Representations, ICLR '15, 2015.

[28]

Wouter Kool, Herke Van Hoof, and Max Welling. Stochastic beams and where to find them: The gumbel-top-k trick for sampling sequences without replacement. In International Conference on Machine Learning, pages 3499--3508. PMLR, 2019.

[29]

Wouter Kool, Herke van Hoof, and Max Welling. Ancestral gumbel-top-k sampling for sampling without replacement. J. Mach. Learn. Res., 21:47--1, 2020.

[30]

Yen-Chieh Lien, Daniel Cohen, andWBruce Croft. An assumption-free approach to the dynamic truncation of ranked lists. In Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval, pages 79--82, 2019.

Digital Library

[31]

Tie-Yan Liu. Learning to rank for information retrieval. 2011.

[32]

Craig Macdonald, Rodrygo L. Santos, and Iadh Ounis. The whens and hows of learning to rank for web search. Inf. Retr., 16(5):584--628, oct 2013.

Digital Library

[33]

Craig Macdonald and Nicola Tonellotto. Declarative experimentation ininformation retrieval using pyterrier. In Proceedings of ICTIR 2020, 2020.

[34]

Joel Mackenzie, J. Shane Culpepper, Roi Blanco, Matt Crane, Charles L. A. Clarke, and Jimmy Lin. Query driven algorithm selection in early stage retrieval. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining, WSDM '18, page 396--404, New York, NY, USA, 2018. Association for Computing Machinery.

Digital Library

[35]

Chris J. Maddison, Andriy Mnih, and Yee Whye Teh. The concrete distribution: A continuous relaxation of discrete random variables. In Proceedings of the 5th International Conference on Learning Representations, ICLR '17, 2017.

[36]

Raghavan Manmatha, Toni Rath, and Fangfang Feng. Modeling score distributions for combining the outputs of search engines. In Proceedings of the 24th annual international ACM SIGIR conference on Research and development in information retrieval, pages 267--275, 2001.

Digital Library

[37]

Ramesh Nallapati. Discriminative models for information retrieval. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '04, page 64--71, New York, NY, USA, 2004. Association for Computing Machinery.

Digital Library

[38]

Rodrigo Nogueira and Kyunghyun Cho. Passage re-ranking with BERT. CoRR, 2019.

[39]

Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with a pretrained sequence-to-sequence model. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 708--718, Online, November 2020. Association for Computational Linguistics.

[40]

Rama Kumar Pasumarthi, Sebastian Bruch, Xuanhui Wang, Cheng Li, Michael Bendersky, Marc Najork, Jan Pfeifer, Nadav Golbandi, Rohan Anil, and Stephan Wolf. Tf-ranking: Scalable tensorflow library for learning-to-rank. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2970--2978, 2019.

Digital Library

[41]

Joaquín Pérez-Iglesias and Lourdes Araujo. Standard deviation as a query hardness estimator. In Proceedings of the 17th International Conference on String Processing and Information Retrieval, SPIRE'10, page 207--212, Berlin, Heidelberg, 2010. Springer-Verlag.

Digital Library

[42]

R. L. Plackett. The analysis of permutations. Journal of the Royal Statistical Society. Series C (Applied Statistics), 24(2):193--202, 1975.

[43]

Jay M. Ponte and W. Bruce Croft. A language modeling approach to information retrieval. In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '98, page 275--281, New York, NY, USA, 1998. Association for Computing Machinery.

Digital Library

[44]

Prafull Prakash, Julian Killingback, and Hamed Zamani. Learning robust dense retrieval models from incomplete relevance labels. In Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '21, page 1728--1732, New York, NY, USA, 2021. Association for Computing Machinery.

Digital Library

[45]

Yingqi Qu, Yuchen Ding, Jing Liu, Kai Liu, Ruiyang Ren, Wayne Xin Zhao, Daxiang Dong, Hua Wu, and Haifeng Wang. RocketQA: An optimized training approach to dense passage retrieval for open-domain question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5835--5847, Online, June 2021. Association for Computational Linguistics.

[46]

Ruiyang Ren, Yingqi Qu, Jing Liu, Wayne Xin Zhao, Qiaoqiao She, Hua Wu, Haifeng Wang, and Ji-Rong Wen. Rocketqav2: A joint training method for dense passage retrieval and passage re-ranking. CoRR, abs/2110.07367, 2021.

[47]

S.E. Robertson, S. Walker, S. Jones, M.M. Hancock-Beaulieu, and M. Gatford. Okapi at trec-3. In TREC '96, pages 109--126, Gaithersburg, Maryland, USA, 1996.

[48]

Stephen Robertson, Evangelos Kanoulas, and Emine Yilmaz. Modelling score distributions without actual scores. In Proceedings of the 2013 Conference on the Theory of Information Retrieval, pages 85--92, 2013.

Digital Library

[49]

Haggai Roitman, Shay Hummel, and Oren Kurland. Using the cross-entropy method to re-rank search results. In Proceedings of the 37th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '14, page 839--842, New York, NY, USA, 2014. Association for Computing Machinery.

Digital Library

[50]

Gerard Salton and Christopher Buckley. Term-weighting approaches in automatic text retrieval. Inf. Process. Manage., 24(5):513--523, aug 1988.

Digital Library

[51]

Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. Colbertv2: Effective and efficient retrieval via lightweight late interaction. CoRR, abs/2112.01488, 2021.

[52]

Markus Schedl, Hamed Zamani, Ching-Wei Chen, Yashar Deldjoo, and Mehdi Elahi. Current challenges and visions in music recommender systems research. Int. J. Multim. Inf. Retr., 7(2):95--116, 2018.

[53]

Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. Predicting query performance by query-drift estimation. ACM Trans. Inf. Syst., 30(2), may 2012.

[54]

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In NeurIPS '17, 2017.

[55]

Lidan Wang, Jimmy Lin, and Donald Metzler. A cascade ranking model for efficient ranked retrieval. In Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval, pages 105--114, 2011.

Digital Library

[56]

Fen Xia, Tie-Yan Liu, JueWang,Wensheng Zhang, and Hang Li. Listwise approach to learning to rank: theory and algorithm. In Proceedings of the 25th international conference on Machine learning, pages 1192--1199, 2008.

Digital Library

[57]

Dawei Yin, Yuening Hu, Jiliang Tang, Tim Daly, Mianwei Zhou, Hua Ouyang, Jianhui Chen, Changsung Kang, Hongbo Deng, Chikashi Nobata, Jean-Marc Langlois, and Yi Chang. Ranking relevance in yahoo search. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '16, page 323--332, New York, NY, USA, 2016. Association for Computing Machinery.

Digital Library

[58]

Hamed Zamani and W. Bruce Croft. On the theory of weak supervision for information retrieval. In Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval, ICTIR '18, page 147--154, New York, NY, USA, 2018. Association for Computing Machinery.

Digital Library

[59]

Hansi Zeng, Hamed Zamani, and Vishwa Vinay. Curriculum learning for dense retrieval distillation. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '22, New York, NY, USA, 2022. Association for Computing Machinery.

Digital Library

[60]

Yi Zhang and Jamie Callan. Maximum likelihood estimation for filtering thresholds. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '01, page 294--302, New York, NY, USA, 2001. Association for Computing Machinery.

Digital Library

Cited By

Rossi NLin JLiu FYang ZLee TMagnani ALiao CSerra ESpezzano F(2024)Relevance Filtering for Embedding-based RetrievalProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680095(4828-4835)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680095
Zamani HBendersky MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility MaximizationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657923(2641-2646)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657923
Meng CArabzadeh NAskari AAliannejadi Mde Rijke MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Ranked List Truncation for Large Language Model-based Re-RankingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657864(141-151)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657864
Show More Cited By

Index Terms

Stochastic Retrieval-Conditioned Reranking
1. Information systems
  1. Information retrieval
    1. Retrieval models and ranking
      1. Learning to rank

Recommendations

Improving retrieval of plane geometry figure with learning to rank

A learning-to-rank method for PGF retrieve.An embedded feature selection for ranking.Improve efficiency with a feature group technology. Display Omitted Educational images are increasingly becoming available online, but an effective method to search for ...
Metric-agnostic Ranking Optimization
SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Ranking is at the core of Information Retrieval. Classic ranking optimization studies often treat ranking as a sorting problem with the assumption that the best performance of ranking would be achieved if we rank items according to their individual ...
A machine learning approach for improved BM25 retrieval
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Despite the widespread use of BM25, there have been few studies examining its effectiveness on a document description over single and multiple field combinations. We determine the effectiveness of BM25 on various document fields. We find that BM25 ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICTIR '22: Proceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval

August 2022

289 pages

ISBN:9781450394123

DOI:10.1145/3539813

Program Chairs:
Fabio Crestani
Università della Svizzera Italiana - USI, Switzerland
,
Gabriella Pasi
Univ. Milano-Bicocca, Italy
,
Eric Gaussier
Univ. Grenoble-Alpes, France

Copyright © 2022 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 25 August 2022

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICTIR '22

Sponsor:

SIGIR

ICTIR '22: The 2022 ACM SIGIR International Conference on the Theory of Information Retrieval

July 11 - 12, 2022

Madrid, Spain

Acceptance Rates

ICTIR '22 Paper Acceptance Rate 32 of 80 submissions, 40%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
675
Total Downloads

Downloads (Last 12 months)300
Downloads (Last 6 weeks)35

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Rossi NLin JLiu FYang ZLee TMagnani ALiao CSerra ESpezzano F(2024)Relevance Filtering for Embedding-based RetrievalProceedings of the 33rd ACM International Conference on Information and Knowledge Management10.1145/3627673.3680095(4828-4835)Online publication date: 21-Oct-2024
https://dl.acm.org/doi/10.1145/3627673.3680095
Zamani HBendersky MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Stochastic RAG: End-to-End Retrieval-Augmented Generation through Expected Utility MaximizationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657923(2641-2646)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657923
Meng CArabzadeh NAskari AAliannejadi Mde Rijke MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Ranked List Truncation for Large Language Model-based Re-RankingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657864(141-151)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657864
Bruch SGai SIngber A(2023)An Analysis of Fusion Functions for Hybrid RetrievalACM Transactions on Information Systems10.1145/359651242:1(1-35)Online publication date: 18-Aug-2023
https://dl.acm.org/doi/10.1145/3596512

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten