short-paper

Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences

Authors:

Ashutosh Singh,

Debasis Ganguly,

Craig McDonaldAuthors Info & Claims

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 2486 - 2490

https://doi.org/10.1145/3539618.3592082

Published: 18 July 2023 Publication History

Abstract

A query performance prediction (QPP) method predicts the effectiveness of an IR system for a given query. While unsupervised approaches have been shown to work well for statistical IR models, it is likely that these approaches would yield limited effectiveness for neural ranking models (NRMs) because the retrieval scores of these models lie within a short range unlike their statistical counterparts. In this work, we propose to leverage a pairwise inference-based NRM's (specifically, DuoT5) output to accumulate evidences on the pairwise believes of one document ranked above the other. We hypothesize that the more consistent these pairwise likelihoods are, the higher is the likelihood of the retrieval to be of better quality, thus yielding a higher QPP score. We conduct our experiments on the TREC-DL dataset leveraging pairwise likelihoods from an auxiliary model DuoT5. Our experiments demonstrate that the proposed method called Pairwise Rank Preference-based QPP (QPP-PRP) leads to significantly better results than a number of standard unsupervised QPP baselines on several NRMs.

References

[1]

Negar Arabzadeh, Maryam Khodabakhsh, and Ebrahim Bagheri. 2021. BERT-QPP: Contextualized Pre-Trained Transformers for Query Performance Prediction. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management (Virtual Event, Queensland, Australia) (CIKM '21). Association for Computing Machinery, 2857--2861.

Digital Library

[2]

Zhuyun Dai and Jamie Callan. 2020. Context-Aware Document Term Weighting for Ad-Hoc Search. In Proceedings of The Web Conference 2020 (Taipei, Taiwan) (WWW '20). Association for Computing Machinery, 1897--1907.

Digital Library

[3]

Zhuyun Dai, Chenyan Xiong, Jamie Callan, and Zhiyuan Liu. 2018. Convolutional Neural Networks for Soft-Matching N-Grams in Ad-Hoc Search. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (Marina Del Rey, CA, USA) (WSDM '18). Association for Computing Machinery, 126--134.

Digital Library

[4]

Suchana Datta, Debasis Ganguly, Derek Greene, and Mandar Mitra. 2022a. Deep-QPP: A Pairwise Interaction-Based Deep Learning Model for Supervised Query Performance Prediction. In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining (Virtual Event, AZ, USA) (WSDM '22). Association for Computing Machinery, 201--209.

Digital Library

[5]

Suchana Datta, Debasis Ganguly, Mandar Mitra, and Derek Greene. 2022b. A Relative Information Gain-Based Query Performance Prediction Framework with Generated Query Variants. ACM Trans. Inf. Syst., Vol. 41, 2, Article 38 (Dec 2022), 31 pages.

[6]

Suchana Datta, Sean MacAvaney, Debasis Ganguly, and Derek Greene. 2022c. A 'Pointwise-Query, Listwise-Document' based Query Performance Prediction Approach. In In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval (Madrid, Spain) (SIGIR '22). ACM, New York, NY, USA, 2148--2153.

Digital Library

[7]

Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural Ranking Models with Weak Supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (Shinjuku, Tokyo, Japan) (SIGIR '17). Association for Computing Machinery, New York, NY, USA, 65--74.

Digital Library

[8]

Jiafeng Guo, Yixing Fan, Qingyao Ai, and W. Bruce Croft. 2016. A Deep Relevance Matching Model for Ad-Hoc Retrieval. In Proceedings of the 25th ACM International on Conference on Information and Knowledge Management (CIKM '16). ACM, New York, NY, USA, 55--64.

[9]

Claudia Hauff. 2010. Predicting the Effectiveness of Queries and Retrieval Systems. SIGIR Forum, Vol. 44, 1 (Aug. 2010), 88.

Digital Library

[10]

Yongyu Jiang, Peng Zhang, Hui Gao, and Dawei Song. 2020. A Quantum Interference Inspired Neural Matching Model for Ad-Hoc Retrieval. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, 19--28.

Digital Library

[11]

Omar Khattab and Matei Zaharia. 2020. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (Virtual Event, China) (SIGIR '20). Association for Computing Machinery, 39--48.

Digital Library

[12]

Victor Lavrenko and W. Bruce Croft. 2001. Relevance Based Language Models. In Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (New Orleans, Louisiana, USA) (SIGIR '01). Association for Computing Machinery, 120--127.

Digital Library

[13]

Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. CoRR, Vol. abs/1611.09268 (2016).

[14]

Rodrigo Frassetto Nogueira, Wei Yang, Jimmy Lin, and Kyunghyun Cho. 2019. Document Expansion by Query Prediction. CoRR, Vol. abs/1904.08375 (2019).

[15]

Ronak Pradeep, Rodrigo Nogueira, and Jimmy Lin. 2021. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models. CoRR, Vol. abs/2101.05667 (2021).

[16]

Dwaipayan Roy, Debasis Ganguly, Mandar Mitra, and Gareth J. F. Jones. 2019. Estimating Gaussian mixture models in the local neighbourhood of embedded word vectors for query performance prediction. Inf. Process. Manag., Vol. 56, 3 (2019), 1026--1045.

Digital Library

[17]

Anna Shtok, Oren Kurland, and David Carmel. 2010. Using Statistical Decision Theory and Relevance Models for Query-Performance Prediction. In Proceedings of the 33rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '10). Association for Computing Machinery, 259--266.

Digital Library

[18]

Anna Shtok, Oren Kurland, David Carmel, Fiana Raiber, and Gad Markovits. 2012. Predicting Query Performance by Query-Drift Estimation. ACM Trans. Inf. Syst., Vol. 30, 2, Article 11 (2012), 35 pages.

Digital Library

[19]

Xiao Wang, Craig MacDonald, Nicola Tonellotto, and Iadh Ounis. 2023. ColBERT-PRF: Semantic Pseudo-Relevance Feedback for Dense Passage and Document Retrieval. ACM Trans. Web, Vol. 17, 1, Article 3 (Jan 2023), 39 pages.

Digital Library

[20]

William Webber, Alistair Moffat, and Justin Zobel. 2010. A Similarity Measure for Indefinite Rankings. ACM Trans. Inf. Syst., Vol. 28, 4, Article 20 (Nov 2010), 38 pages. https://doi.org/10.1145/1852102.1852106

Digital Library

[21]

Chenyan Xiong, Zhuyun Dai, Jamie Callan, Zhiyuan Liu, and Russell Power. 2017. End-to-End Neural Ad-Hoc Ranking with Kernel Pooling. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '17). ACM, 55--64.

Digital Library

[22]

Lee Xiong, Chenyan Xiong, Ye Li, Kwok-Fung Tang, Jialin Liu, Paul N. Bennett, Junaid Ahmed, and Arnold Overwijk. 2020. Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrieval. CoRR, Vol. abs/2007.00808 (2020).

[23]

Hamed Zamani, W. Bruce Croft, and J. Shane Culpepper. 2018. Neural Query Performance Prediction Using Weak Supervision from Multiple Signals. In The 41st International ACM SIGIR Conference on Research and Development in Information Retrieval (Ann Arbor, MI, USA) (SIGIR '18). ACM, New York, NY, USA, 105--114.

[24]

Oleg Zendel, Anna Shtok, Fiana Raiber, Oren Kurland, and J. Shane Culpepper. 2019. Information Needs, Queries, and Query Performance Prediction. In Proc. of SIGIR '19 (Paris, France) (SIGIR'19). Association for Computing Machinery, 395--404.

[25]

Zhi Zheng, Kai Hui, Ben He, Xianpei Han, Le Sun, and Andrew Yates. 2020. BERT-QE: Contextualized Query Expansion for Document Re-ranking. In Findings of the Association for Computational Linguistics: EMNLP 2020. Association for Computational Linguistics, Online, 4718--4728.

[26]

Yun Zhou and W. Bruce Croft. 2007. Query Performance Prediction in Web Search Environments. In Proc. 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '07). Association for Computing Machinery, 543--550.

Cited By

Parry AGanguly DChandra MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657842(14-25)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657842
Arabzadeh NMeng CAliannejadi MBagheri E(2024)Query Performance Prediction: From Fundamentals to Advanced TechniquesAdvances in Information Retrieval10.1007/978-3-031-56069-9_51(381-388)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56069-9_51
Ebrahimi SKhodabakhsh MArabzadeh NBagheri E(2024)Estimating Query Performance Through Rich Contextualized Query RepresentationsAdvances in Information Retrieval10.1007/978-3-031-56066-8_6(49-58)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56066-8_6
Show More Cited By

Index Terms

Unsupervised Query Performance Prediction for Neural Models with Pairwise Rank Preferences
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing
      1. Query intent

Recommendations

A 'Pointwise-Query, Listwise-Document' based Query Performance Prediction Approach
SIGIR '22: Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval

The task of Query Performance Prediction (QPP) in Information Retrieval (IR) involves predicting the relative effectiveness of a search system for a given input query. Supervised approaches for QPP, such as NeuralQPP are often trained on pairs of ...
Neural Query Performance Prediction using Weak Supervision from Multiple Signals
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Predicting the performance of a search engine for a given query is a fundamental and challenging task in information retrieval. Accurate performance predictors can be used in various ways, such as triggering an action, choosing the most effective ...
Query-performance prediction: setting the expectations straight
SIGIR '14: Proceedings of the 37th international ACM SIGIR conference on Research & development in information retrieval

The query-performance prediction task has been described as estimating retrieval effectiveness in the absence of relevance judgments. The expectations throughout the years were that improved prediction techniques would translate to improved retrieval ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '23: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 2023

3567 pages

ISBN:9781450394086

DOI:10.1145/3539618

General Chairs:
Hsin-Hsi Chen
National Taiwan University
,
Wei-Jou (Edward) Duh
National Taiwan University
,
Hen-Hsen Huang
Academia Sinica
,
Program Chairs:
Makoto P. Kato
Spotify
,
Josiane Mothe
Universite de Toulouse
,
Barbara Poblete
University of Chile and Amazon Visiting Academic

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 July 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Science Foundation Ireland

Conference

SIGIR '23

Sponsor:

SIGIR

SIGIR '23: The 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

July 23 - 27, 2023

Taipei, Taiwan

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
168
Total Downloads

Downloads (Last 12 months)84
Downloads (Last 6 weeks)0

Reflects downloads up to 18 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Parry AGanguly DChandra MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)"In-Context Learning" or: How I learned to stop worrying and love "Applied Information Retrieval"Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657842(14-25)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657842
Arabzadeh NMeng CAliannejadi MBagheri E(2024)Query Performance Prediction: From Fundamentals to Advanced TechniquesAdvances in Information Retrieval10.1007/978-3-031-56069-9_51(381-388)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56069-9_51
Ebrahimi SKhodabakhsh MArabzadeh NBagheri E(2024)Estimating Query Performance Through Rich Contextualized Query RepresentationsAdvances in Information Retrieval10.1007/978-3-031-56066-8_6(49-58)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56066-8_6
Khodabakhsh MZarrinkalam FArabzadeh N(2024)BertPE: A BERT-Based Pre-retrieval Estimator for Query Performance PredictionAdvances in Information Retrieval10.1007/978-3-031-56063-7_27(354-363)Online publication date: 24-Mar-2024
https://dl.acm.org/doi/10.1007/978-3-031-56063-7_27

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents