research-article

Visual Pool: A Tool to Visualize and Interact with the Pooling Method

Authors:

Allan HanburyAuthors Info & Claims

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 1321 - 1324

https://doi.org/10.1145/3077136.3084146

Published: 07 August 2017 Publication History

Abstract

Every year more than 25 test collections are built among the main Information Retrieval (IR) evaluation campaigns. They are extremely important in IR because they become the evaluation praxis for the forthcoming years. Test collections are built mostly using the pooling method. The main advantage of this method is that it drastically reduces the number of documents to be judged. It does so at the cost of introducing biases, which are sometimes aggravated by non optimal configuration. In this paper we develop a novel visualization technique for the pooling method, and integrate it in a demo application named Visual Pool. This demo application enables the user to interact with the pooling method with ease, and develops visual hints in order to analyze existing test collections, and build better ones.

References

[1]

Javed A. Aslam and Mark Montague 2001. Models for Metasearch Proc. of SIGIR.

[2]

Gordon V. Cormack, Charles L. A. Clarke, and Stefan Buettcher. 2009. Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods Proc. of SIGIR.

[3]

Gordon V. Cormack, Christopher R. Palmer, and Charles L. A. Clarke 1998. Efficient Construction of Large Test Collections. Proc. of SIGIR.

Digital Library

[4]

Bevan Koopman and Guido Zuccon 2014. Why assessing relevance in medical IR is demanding Medical Information Retrieval (MedIR) Workshop.

[5]

Aldo Lipani. 2016. Fairness in Information Retrieval. In Proc. of SIGIR.

Digital Library

[6]

Aldo Lipani, Mihai Lupu, and Allan Hanbury 2015. Splitting Water: Precision and Anti-Precision to Reduce Pool Bias Proc. of SIGIR.

[7]

Aldo Lipani, Mihai Lupu, and Allan Hanbury 2016. The Curious Incidence of Bias Corrections in the Pool Proc. of ECIR.

[8]

Aldo Lipani, Mihai Lupu, Evangelos Kanoulas, and Allan Hanbury. 2016. The Solitude of Relevant Documents in the Pool. In Proc. of CIKM.

Digital Library

[9]

Aldo Lipani, Mihai Lupu, Joao Palotti, Guido Zuccon, and Allan Hanbury 2017. Fixed Budget Pooling Strategies Based on Fusion Methods Proc. of SAC.

[10]

Aldo Lipani, Joao Palotti, Mihai Lupu, Florina Piroi, Guido Zuccon, and Allan Hanbury. 2017. Fixed-Cost Pooling Strategies Based on IR Evaluation Measures.

[11]

David E. Losada, Javier Parapar, and Alvaro Barreiro. 2017. Multi-armed bandits for adjudicating documents in pooling-based evaluation of information retrieval systems. Information Processing & Management Vol. 53, 5 (2017).

[12]

Craig Macdonald and Iadh Ounis 2006. Voting for Candidates: Adapting Data Fusion Techniques for an Expert Search Task Proc. of CIKM.

[13]

Alistair Moffat, William Webber, and Justin Zobel. 2007. Strategic System Comparisons via Targeted Relevance Judgments Proc. of SIGIR.

[14]

K. Spärck Jones and C. J. van Rijsbergen 1975. Report on the need for and provision of an 'ideal' information retrieval test collection. British Library Research and Development Report No. 5266 (1975).

[15]

Ellen M. Voorhees. 2014. The Effect of Sampling Strategy on Inferred Measures Proc. of SIGIR.

[16]

E. Voorhes and Donna Harman 1999. Overview of the eighth text retrieval conference. Proc. of TREC.

[17]

Emine Yilmaz and Javed A. Aslam 2006. Estimating Average Precision with Incomplete and Imperfect Judgments Proc. of CIKM.

Cited By

Hofstätter SZlabinger MHanbury A(2020)Neural-IR-Explorer: A Content-Focused Tool to Explore Neural Re-ranking ResultsAdvances in Information Retrieval10.1007/978-3-030-45442-5_58(459-464)Online publication date: 8-Apr-2020
https://doi.org/10.1007/978-3-030-45442-5_58
Rocco GSilvello G(2020)An Information Visualization Tool for the Interactive Component-Based Evaluation of Search EnginesDigital Libraries: The Era of Big Data and Data Science10.1007/978-3-030-39905-4_3(15-25)Online publication date: 22-Jan-2020
https://doi.org/10.1007/978-3-030-39905-4_3
Palotti JScells HZuccon GPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)TrecToolsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331399(1325-1328)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331399
Show More Cited By

Index Terms

Visual Pool: A Tool to Visualize and Interact with the Pooling Method
1. Human-centered computing
  1. Visualization
    1. Visualization techniques
2. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Relevance assessment
      2. Test collections

Recommendations

Fairness in Information Retrieval
SIGIR '16: Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval

The offline evaluation of Information Retrieval (IR) systems is performed through the use of test collections. A test collection, in its essence, is composed of: a collection of documents, a set of topics and, a set of relevance assessments for each ...
Fixed budget pooling strategies based on fusion methods
SAC '17: Proceedings of the Symposium on Applied Computing

The empirical nature of Information Retrieval (IR) mandates strong experimental practices. The Cranfield/TREC evaluation paradigm represents a keystone of such experimental practices. Within this paradigm, the generation of relevance judgments has been ...
The Impact of Fixed-Cost Pooling Strategies on Test Collection Bias
ICTIR '16: Proceedings of the 2016 ACM International Conference on the Theory of Information Retrieval

In Information Retrieval, test collections are usually built using the pooling method. Many pooling strategies have been developed for the pooling method. Herein, we address the question of identifying the best pooling strategy when evaluating systems ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

August 2017

1476 pages

ISBN:9781450350228

DOI:10.1145/3077136

General Chairs:
Noriko Kando
National Institute of Informatics
,
Tetsuya Sakai
Waseda University
,
Hideo Joho
University of Tsukuba
,
Program Chairs:
Hang Li
Huawei Noah's Ark Lab
,
Arjen P. de Vries
Radboud University
,
Ryen W. White
Microsoft Cortana

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Austrian Science Fund (FWF)

Conference

SIGIR '17

Sponsor:

SIGIR

SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval

August 7 - 11, 2017

Tokyo, Shinjuku, Japan

Acceptance Rates

SIGIR '17 Paper Acceptance Rate 78 of 362 submissions, 22%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
170
Total Downloads

Downloads (Last 12 months)7
Downloads (Last 6 weeks)2

Reflects downloads up to 05 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Hofstätter SZlabinger MHanbury A(2020)Neural-IR-Explorer: A Content-Focused Tool to Explore Neural Re-ranking ResultsAdvances in Information Retrieval10.1007/978-3-030-45442-5_58(459-464)Online publication date: 8-Apr-2020
https://doi.org/10.1007/978-3-030-45442-5_58
Rocco GSilvello G(2020)An Information Visualization Tool for the Interactive Component-Based Evaluation of Search EnginesDigital Libraries: The Era of Big Data and Data Science10.1007/978-3-030-39905-4_3(15-25)Online publication date: 22-Jan-2020
https://doi.org/10.1007/978-3-030-39905-4_3
Palotti JScells HZuccon GPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)TrecToolsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331399(1325-1328)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331399
Lipani A(2019)On Biases in Information Retrieval Models and EvaluationACM SIGIR Forum10.1145/3308774.330880452:2(172-173)Online publication date: 17-Jan-2019
https://dl.acm.org/doi/10.1145/3308774.3308804
Ferro NSantucci G(2019)Visual Analytics and IR Experimental EvaluationInformation Retrieval Evaluation in a Changing World10.1007/978-3-030-22948-1_24(565-582)Online publication date: 14-Aug-2019
https://doi.org/10.1007/978-3-030-22948-1_24

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents