research-article

A Formal Account of Effectiveness Evaluation and Ranking Fusion

Authors:

Enrique Amigó,

Fernando Giner,

Stefano Mizzaro,

Damiano SpinaAuthors Info & Claims

ICTIR '18: Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval

Pages 123 - 130

https://doi.org/10.1145/3234944.3234958

Published: 10 September 2018 Publication History

Abstract

This paper proposes a theoretical framework which models the information provided by retrieval systems in terms of Information Theory. The proposed framework allows to formalize: (i) system effectiveness as an information theoretic similarity between system outputs and human assessments, and (ii) ranking fusion as an information quantity measure. As a result, the proposed effectiveness metric improves popular metrics in terms of formal constraints. In addition, our empirical experiments suggest that it captures quality aspects from traditional metrics, while the reverse is not true. Our work also advances the understanding of theoretical foundations of the empirically known phenomenon of effectiveness increase when combining retrieval system outputs in an unsupervised manner.

References

[1]

Enrique Amigó, Fernando Giner, Julio Gonzalo, and Felisa Verdejo . 2017. A Formal and Empirical Study of Unsupervised Signal Combination for Textual Similarity Tasks. In Proceedings of ECIR'17. 369--382.

[2]

Enrique Amigó, Fernando Giner, Julio Gonzalo, and Felisa Verdejo . 2017. An Axiomatic Account of Similarity. In Proceedings of the SIGIR'17 Workshop on Axiomatic Thinking for Information Retrieval and Related Tasks (ATIR).

[3]

Enrique Amigó, Julio Gonzalo, and Felisa Verdejo . 2013. A General Evaluation Measure for Document Organization Tasks Proceedings of SIGIR'13. 643--652.

Digital Library

[4]

Enrique Amigó, Damiano Spina, and Jorge Carrillo-de Albornoz . 2018. An Axiomatic Analysis of Diversity Evaluation Metrics: Introducing the Rank-Biased Utility Metric. In Proceedings of SIGIR'18. 625--634.

Digital Library

[5]

Yael Anava, Anna Shtok, Oren Kurland, and Ella Rabinovich . 2016. A Probabilistic Fusion Framework. In Proceedings of CIKM'16. 1463--1472.

Digital Library

[6]

Javed A. Aslam and Robert Savell . 2003. On the Effectiveness of Evaluating Retrieval Systems in the Absence of Relevance Judgments. In Proceedings of SIGIR'03. 361--362.

Digital Library

[7]

David Austen-Smith and Jeffrey S. Banks . 1996. Information Aggregation, Rationality, and the Condorcet Jury Theorem. American Political Science Review Vol. 90, 1 (1996), 34--45.

[8]

Ashraf Bah Rabiou and Ben Carterette . 2016. A Probabilistic Data Fusion Framework for Retrieval and Ranking Proceedings of ICTIR'16. 31--39.

Digital Library

[9]

Luca Busin and Stefano Mizzaro . 2013. Axiometrics: An Axiomatic Approach to Information Retrieval Effectiveness Metrics Proceedings of ICTIR'13. 22--29.

Digital Library

[10]

Olivier Chapelle, Donald Metzler, Ya Zhang, and Pierre Grinspan . 2009. Expected Reciprocal Rank for Graded Relevance. In Proceedings of CIKM'09. 621--630.

Digital Library

[11]

Charles L.A. Clarke, Nick Craswell, and Ian Soboroff . 2004. Overview of the TREC 2004 Terabyte Track. In Proceedings of TREC'04.

[12]

Marquis de Condorcet . 1785. Essay on the Application of Analysis to the Probability of Majority Decisions. Paris: Imprimerie Royale (. 1785).

[13]

Wagner Teixeira da Silva and Ruy Luiz Milidiú . 1993. Belief Function Model for information retrieval. Journal of the American Society for Information Science Vol. 44, 1 (1993), 10--18.

[14]

Arthur P. Dempster . 1967. Upper and Lower Probabilities Induced by a Multivalued Mapping. The Annals of Mathematical Statistics (1967), 325--339.

[15]

Marco Ferrante, Nicola Ferro, and Maria Maistro . 2015. Towards a Formal Framework for Utility-oriented Measurements of Retrieval Effectiveness. In Proceedings of ICTIR'15. New York, NY, USA, 21--30.

Digital Library

[16]

Marco Ferrante, Nicola Ferro, and Silvia Pontarollo . 2018. A General Theory of IR Evaluation Measures. IEEE Trans. on Knowl. and Data Eng. (2018).

[17]

Peter B. Golbus, Javed A. Aslam, and Charles L. A. Clarke . 2013. Increasing Evaluation Sensitivity to Diversity. Inf. Retr. Vol. 16, 4 (2013), 530--555.

Digital Library

[18]

Lars K. Hansen and Peter Salamon . 1990. Neural Network Ensembles. IEEE Trans. Pattern Anal. Mach. Intell. Vol. 12, 10 (1990), 993--1001.

Digital Library

[19]

David A. Hull, Jan O. Pedersen, and Hinrich Schütze . 1996. Method Combination for Document Filtering. In Proceedings of SIGIR'96. 279--287.

Digital Library

[20]

Kalervo J"arvelin and Jaana Kek"al"ainen . 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM Trans. Inf. Syst. Vol. 20 (2002), 422--446.

Digital Library

[21]

Serguei Kaniovski and Alexander Zaigraev . 2011. Optimal Jury Design for Homogeneous Juries with Correlated Votes. Theory and Decision Vol. 71, 4 (2011), 439--459.

[22]

Anders Krogh and Jesper Vedelsby . 1995. Neural Network Ensembles, Cross Validation, and Active Learning Proceedings of NIPS'95. 231--238.

Digital Library

[23]

Mounia Lalmas and Ian Ruthven . 1998. Representing and Retrieving Structured Documents using the Dempster-Shafer Theory of Evidence: Modelling and Evaluation. Journal of Documentation Vol. 54, 5 (1998), 529--565.

[24]

Joon Ho Lee . 1995. Combining Multiple Evidence from Different Properties of Weighting Schemes Proceedings of SIGIR'95. 180--188.

Digital Library

[25]

Gad Markovits, Anna Shtok, Oren Kurland, and David Carmel . 2012. Predicting Query Performance for Fusion-based Retrieval Proceedings of CIKM'12. 813--822.

Digital Library

[26]

Ofer Matan . 1996. On Voting Ensembles of Classifiers. In Proceedings of AAAI-96 Workshop on Integrating Multiple Learned Models. 84--88.

[27]

Alistair Moffat and Justin Zobel . 2008. Rank-Biased Precision for Measurement of Retrieval Effectiveness. ACM Trans. Inf. Syst. Vol. 27, 1, Article bibinfoarticleno2 (2008), bibinfonumpages2:1--2:27 pages.

Digital Library

[28]

Mark H. Montague and Javed A. Aslam . 2002. Condorcet Fusion for Improved Retrieval. In Proceedings of CIKM'02. 538--548.

Digital Library

[29]

Rabia Nuray-Turan and Fazli Can . 2006. Automatic Ranking of Retrieval Systems using Fusion Data. Vol. 42 (05 . 2006), 595--614.

Digital Library

[30]

Glenn Shafer . 1976. A Mathematical Theory of Evidence. Vol. Vol. 42. Princeton University Press.

[31]

Claude E. Shannon . 1948. A Mathematical Theory of Communication. The Bell System Technical Journal Vol. 27, 3 (7 . 1948), 379--423.

[32]

Joseph A. Shaw, Edward A. Fox, Joseph A. Shaw, and Edward A. Fox . 1994. Combination of Multiple Searches. In Proceedings of TREC-2. 243--252.

[33]

Ian Soboroff, Charles Nicholas, and Patrick Cahan . 2001. Ranking Retrieval Systems without Relevance Judgments Proceedings of SIGIR'01. 66--73.

Digital Library

[34]

Cornelis J. van Rijsbergen . 1979. Information Retrieval (bibinfoedition2nd ed.). Butterworth-Heinemann.

Digital Library

[35]

Christopher C. Vogt and Garrison W. Cottrell . 1998. Predicting the Performance of Linearly Combined IR Systems Proceedings of SIGIR'98. 190--196.

Digital Library

[36]

Ellen M. Voorhees . 1999. The TREC-8 Question Answering Track Report. In Proceedings of TREC-8. 77--82.

Cited By

Ye FLi SChua TNgo CKa-Wei Lee RKumar RLauw H(2024)MileCut: A Multi-view Truncation Framework for Legal Case RetrievalProceedings of the ACM Web Conference 202410.1145/3589334.3645349(1341-1349)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645349
Amigó EMizzaro SSpina DAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Ranking InterruptusProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532051(588-598)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532051
Chu ZMao JZhang FLiu YSakai TZhang MMa SDemartini GZuccon GCulpepper JHuang ZTong H(2021)Evaluating Relevance Judgments with Pairwise Discriminative PowerProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482428(261-270)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482428
Show More Cited By

Index Terms

A Formal Account of Effectiveness Evaluation and Ranking Fusion
1. Information systems
  1. Information retrieval

Recommendations

Effective Ranking Fusion Methods for Personalized Metasearch Engines
PCI '08: Proceedings of the 2008 Panhellenic Conference on Informatics

Metasearch engines are a significant part of the information retrieval process. Most of Web users use them directly or indirectly to access information from more than one data sources. The cornerstone of their technology is their rank aggregation method,...
A Political and Historical Account of Lower Canada: With Remarks on the Present Situation of the People, as Regards Their Manners, Character, Religion, &C. &C
A Brief Account of the Life, Writings, and Inventions of Sir Samuel Morland: Master of Mechanics to Charles the Second

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICTIR '18: Proceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval

September 2018

238 pages

ISBN:9781450356565

DOI:10.1145/3234944

General Chairs:
Dawei Song
Beijing Institute of Technology, China and The Open University, UK
,
Tie-Yan Liu
Microsoft Research Asia, China
,
Le Sun
Chinese Information Processing Society of China, China
,
Program Chairs:
Peter Bruza
Queensland University of Technology, Australia
,
Massimo Melucci
University of Padua, Italy
,
Fabrizio Sebastiani
Consiglio Nazionale delle Ricerche, Italy
,
Grace Hui Yang
Georgetown University, USA

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 September 2018

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Spanish Government
Australian Research Council

Conference

ICTIR '18

Sponsor:

SIGIR

ICTIR '18: The 2018 ACM SIGIR International Conference on the Theory of Information Retrieval

September 14 - 17, 2018

Tianjin, China

Acceptance Rates

ICTIR '18 Paper Acceptance Rate 19 of 47 submissions, 40%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
100
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 02 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Ye FLi SChua TNgo CKa-Wei Lee RKumar RLauw H(2024)MileCut: A Multi-view Truncation Framework for Legal Case RetrievalProceedings of the ACM Web Conference 202410.1145/3589334.3645349(1341-1349)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645349
Amigó EMizzaro SSpina DAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)Ranking InterruptusProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3532051(588-598)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3532051
Chu ZMao JZhang FLiu YSakai TZhang MMa SDemartini GZuccon GCulpepper JHuang ZTong H(2021)Evaluating Relevance Judgments with Pairwise Discriminative PowerProceedings of the 30th ACM International Conference on Information & Knowledge Management10.1145/3459637.3482428(261-270)Online publication date: 26-Oct-2021
https://dl.acm.org/doi/10.1145/3459637.3482428
Giner FAmigó EVerdejo F(2019)Integrating learned and explicit document features for reputation monitoring in social mediaKnowledge and Information Systems10.1007/s10115-019-01383-wOnline publication date: 19-Jul-2019
https://doi.org/10.1007/s10115-019-01383-w

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten