short-paper

Sub-corpora Impact on System Effectiveness

Authors:

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

Pages 901 - 904

https://doi.org/10.1145/3077136.3080674

Published: 07 August 2017 Publication History

Get Access

Abstract

Understanding the factors comprising IR system effectiveness is of primary importance to compare different IR systems. Effectiveness is traditionally broken down, using ANOVA, into a topic and a system effect but this leaves out a key component of our evaluation paradigm: the collections of documents. We break down effectiveness into topic, system and sub-corpus effects and compare it to the traditional break down, considering what happens when different evaluation measures come into play. We found that sub-corpora are a significant effect. The consideration of which allows us to be more accurate in estimating what systems are significantly different. We also found that the sub-corpora affect different evaluation measures in different ways and this may impact on what systems are considered significantly different.

References

[1]

D. Banks, P. Over, and N.-F. Zhang 1999. Blind Men and Elephants: Six Approaches to TREC data. Information Retrieval Vol. 1, 1--2, 7--34.

Digital Library

Google Scholar

[2]

O. Chapelle, D. Metzler, Y. Zhang, and P. Grinspan. 2009. Expected Reciprocal Rank for Graded Relevance. CIKM 2009. 621--630.

Digital Library

Google Scholar

[3]

N. Ferro and G. Silvello. 2016. A General Linear Mixed Models Approach to Study System Component Effects. SIGIR 2016. 25--34.

Digital Library

Google Scholar

[4]

N. Ferro, G. Silvello, H. Keskustalo, A. Pirkola, and K. Järvelin 2016. The Twist Measure for IR Evaluation: Taking User's Effort Into Account. JASIST, Vol. 67, 3, 620--648.

Digital Library

Google Scholar

[5]

Y. Hochberg and A. C. Tamhane. 1987. Multiple Comparison Procedures. John Wiley & Sons, USA.

Crossref

Google Scholar

[6]

K. Järvelin and J. Kekäläinen. 2002. Cumulated Gain-Based Evaluation of IR Techniques. ACM TOIS, Vol. 20, 4, 422--446.

Digital Library

Google Scholar

[7]

T. Jones, A. Turpin, S. Mizzaro, F. Scholer, and M. Sanderson. 2014. Size and Source Matter: Understanding Inconsistencies in Test Collection-Based Evaluation. In CIKM 2014. 1843--1846.

Google Scholar

[8]

A. Moffat and J. Zobel. 2008. Rank-biased Precision for Measurement of Retrieval Effectiveness. ACM TOIS, Vol. 27, 1, 2:1--2:27.

Google Scholar

[9]

S. Olejnik and J. Algina. 2003. Generalized Eta and Omega Squared Statistics: Measures of Effect Size for Some Common Research Designs. Psychological Methods Vol. 8, 4, 434--447.

Crossref

Google Scholar

[10]

S. E. Robertson and E. Kanoulas. 2012. On Per-topic Variance in IR Evaluation. In SIGIR 2012. 891--900.

Digital Library

Google Scholar

[11]

A. Rutherford. 2011. ANOVA and ANCOVA. A GLM Approach (2nd ed.). John Wiley & Sons, New York, USA.

Crossref

Google Scholar

[12]

T. Sakai. 2014. Statistical Reform in Information Retrieval? SIGIR Forum, Vol. 48, 1, 3--12.

Digital Library

Google Scholar

[13]

M. Sanderson, A. Turpin, Y. Zhang, and F. Scholer. 2012. Differences in Effectiveness Across Sub-collections. CIKM 2012. 1965--1969.

Digital Library

Google Scholar

[14]

J. M. Tague-Sutcliffe and J. Blustein. 1994. A Statistical Analysis of the TREC-3 Data. In TREC-3. 385--398.

Google Scholar

Cited By

View all

Rashidi LZobel JMoffat A(2023)The Impact of Judgment Variability on the Consistency of Offline Effectiveness MeasuresACM Transactions on Information Systems10.1145/359651142:1(1-31)Online publication date: 18-Aug-2023
https://dl.acm.org/doi/10.1145/3596511
Rashidi LZobel JMoffat ADiaz FShah CSuel TCastells PJones RSakai T(2021)Evaluating the Predictivity of IR ExperimentsProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463040(1667-1671)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463040
González-Sáez GMulhem PGoeuriot L(2021)Towards the Evaluation of Information Retrieval Systems on Evolving Datasets with Pivot SystemsExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-030-85251-1_8(91-102)Online publication date: 14-Sep-2021
https://doi.org/10.1007/978-3-030-85251-1_8
Show More Cited By

Index Terms

Sub-corpora Impact on System Effectiveness
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Test collections

Recommendations

Using Collection Shards to Study Retrieval Performance Effect Sizes

Despite the bulk of research studying how to more accurately compare the performance of IR systems, less attention is devoted to better understanding the different factors that play a role in such performance and how they interact. This is the case of ...
Information retrieval on Turkish texts

In this study, we investigate information retrieval (IR) on Turkish texts using a large-scale test collection that contains 408,305 documents and 72 ad hoc queries. We examine the effects of several stemming options and query-document matching functions ...
Algorithmic stemmers or morphological analysis? An evaluation

It is important in information retrieval (IR), information extraction, or classification tasks that morphologically related forms are conflated under the same stem (using stemmer) or lemma (using morphological analyzer). To achieve this for the English ...

Comments

Information & Contributors

Information

Published In

SIGIR '17: Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval

August 2017

1476 pages

ISBN:9781450350228

DOI:10.1145/3077136

General Chairs:
Noriko Kando
National Institute of Informatics
,
Tetsuya Sakai
Waseda University
,
Hideo Joho
University of Tsukuba
,
Program Chairs:
Hang Li
Huawei Noah's Ark Lab
,
Arjen P. de Vries
Radboud University
,
Ryen W. White
Microsoft Cortana

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 07 August 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

Conference

SIGIR '17

Sponsor:

SIGIR

SIGIR '17: The 40th International ACM SIGIR conference on research and development in Information Retrieval

August 7 - 11, 2017

Tokyo, Shinjuku, Japan

Acceptance Rates

SIGIR '17 Paper Acceptance Rate 78 of 362 submissions, 22%;

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
160
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)2

Reflects downloads up to 12 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Rashidi LZobel JMoffat A(2023)The Impact of Judgment Variability on the Consistency of Offline Effectiveness MeasuresACM Transactions on Information Systems10.1145/359651142:1(1-31)Online publication date: 18-Aug-2023
https://dl.acm.org/doi/10.1145/3596511
Rashidi LZobel JMoffat ADiaz FShah CSuel TCastells PJones RSakai T(2021)Evaluating the Predictivity of IR ExperimentsProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463040(1667-1671)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463040
González-Sáez GMulhem PGoeuriot L(2021)Towards the Evaluation of Information Retrieval Systems on Evolving Datasets with Pivot SystemsExperimental IR Meets Multilinguality, Multimodality, and Interaction10.1007/978-3-030-85251-1_8(91-102)Online publication date: 14-Sep-2021
https://doi.org/10.1007/978-3-030-85251-1_8
Faggioli GFerro N(2021)System Effect Estimation by Sharding: A Comparison Between ANOVA Approaches to Detect Significant DifferencesAdvances in Information Retrieval10.1007/978-3-030-72240-1_3(33-46)Online publication date: 30-Mar-2021
https://doi.org/10.1007/978-3-030-72240-1_3
Roitero KCarterette BMehrotra RLalmas M(2020)Leveraging Behavioral Heterogeneity Across Markets for Cross-Market Training of Recommender SystemsCompanion Proceedings of the Web Conference 202010.1145/3366424.3384362(694-702)Online publication date: 20-Apr-2020
https://dl.acm.org/doi/10.1145/3366424.3384362
Ferro NSanderson MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Improving the Accuracy of System Performance Estimation by Using ShardsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3338062(805-814)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3338062
Zampieri FRoitero KCulpepper JKurland OMizzaro SPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)On Topic Difficulty in IR EvaluationProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331279(909-912)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331279
Ferro NKim YSanderson M(2019)Using Collection Shards to Study Retrieval Performance Effect SizesACM Transactions on Information Systems10.1145/331036437:3(1-40)Online publication date: 19-Mar-2019
https://dl.acm.org/doi/10.1145/3310364
Ferro NFuhr NGrefenstette GKonstan JCastells PDaly EDeclerck TEkstrand MGeyer WGonzalo JKuflik TLindn KMagnini BNie JPerego RShapira BSoboroff ITintarev NVerspoor KWillemsen MZobel J(2018)The Dagstuhl Perspectives Workshop on Performance Modeling and PredictionACM SIGIR Forum10.1145/3274784.327478952:1(91-101)Online publication date: 31-Aug-2018
https://dl.acm.org/doi/10.1145/3274784.3274789
Ferro NSilvello G(2017)Toward an anatomy of IR system component performancesJournal of the Association for Information Science and Technology10.1002/asi.2391069:2(187-200)Online publication date: 17-Nov-2017
https://doi.org/10.1002/asi.23910

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Using Collection Shards to Study Retrieval Performance Effect Sizes

Information retrieval on Turkish texts

Algorithmic stemmers or morphological analysis? An evaluation