Article

Power and bias of subset pooling strategies

Authors:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 837 - 838

https://doi.org/10.1145/1277741.1277934

Published: 23 July 2007 Publication History

Get Access

Abstract

We define a method to estimate the random and systematic errors resulting from incomplete relevance assessments.Mean Average Precision (MAP) computed over a large number of topics with a shallow assessment pool substantially outperforms -- for the same adjudication effort MAP computed over fewer topics with deeper pools, and P@k computed with pools of the same depth. Move-to-front pooling,previously reported to yield substantially better rank correlation, yields similar power, and lower bias, compared tofixed-depth pooling.

References

[1]

Aslam, J. A., Pavlu, V., and Yilmaz, E. A statistical method for system evaluation using incomplete judgments. In SIGIR '06 (2006), pp. 541--548.

Digital Library

Google Scholar

[2]

Buckley, C., and Voorhees, E. M. Retrieval evaluation with incomplete information. In SIGIR '04 (2004), pp. 25--32.

Digital Library

Google Scholar

[3]

Carterette, B., Allan, J., and Sitaraman, R. Minimal test collections for retrieval evaluation. In SIGIR '06 (2006), pp. 268--275.

Digital Library

Google Scholar

[4]

Cormack, G. V., Palmer, C. R., and Clarke, C. L. A. Efficient construction of large test collections. In SIGIR Conference 1998 (Melbourne, Australia, 1998).

Digital Library

Google Scholar

[5]

Sanderson, M., and Zobel, J. Information retrieval evaluation: Effort, sensitivity, and reliability. In SIGIR Conference 2005 (Salvador, Brazil, 2005).

Digital Library

Google Scholar

[6]

Van Rijsbergen, C. J. Information Retrieval, 2nd edition. Dept. of Computer Science, University of Glasgow, 1979.

Digital Library

Google Scholar

[7]

Voorhees, E. M. Overview of the TREC-2004 robust track. In 13th Text REtrieval Conference (Gaithersburg, MD, 2004).

Google Scholar

Cited By

View all

Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Smirnov PSmith ISafikhani ZBa-alawi WKhodakarami FLin EYu YMartin SOrtmann JAittokallio THafner MHaibe-Kains B(2022)Evaluation of statistical approaches for association testing in noisy drug screening dataBMC Bioinformatics10.1186/s12859-022-04693-z23:1Online publication date: 18-May-2022
https://doi.org/10.1186/s12859-022-04693-z
Otero DParapar JBarreiro ÁHung CHong JBechini ASong E(2021)The wisdom of the rankersProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441947(672-680)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441947
Show More Cited By

Index Terms

Power and bias of subset pooling strategies
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Recommendations

Validity and power of t-test for comparing MAP and GMAP
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

We examine the validity and power of the t-test, Wilcoxon test, and sign test in determining whether or not the difference in performance between two IR systems is significant. Empirical tests conducted on subsets of the TREC2004 Robust Retrieval ...
Hybrid pooling for enhancement of generalization ability in deep convolutional neural networks
Highlights
- Hybrid pooling is proposed for improving the generalization ability of convolutional neural networks.
Abstract
Convolutional neural networks (CNNs) have attracted considerable attention in many application fields for their great ability to deal with image recognition and object detection tasks. A pooling process is an important process in CNNs, ...
Pooling semilattices and non-adaptive pooling designs

In Huang and Weng (2004), Huang and Weng introduced pooling spaces, and constructed pooling designs from a pooling space. In this paper, we introduce the concept of pooling semilattices and prove that a pooling semilattice is a pooling space, then show ...

Comments

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

16
Total Citations
View Citations
342
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Roitero KBarbera DSoprano MDemartini GMizzaro SSakai T(2023)How Many Crowd Workers Do I Need? On Statistical Power when Crowdsourcing Relevance JudgmentsACM Transactions on Information Systems10.1145/359720142:1(1-26)Online publication date: 22-May-2023
https://dl.acm.org/doi/10.1145/3597201
Smirnov PSmith ISafikhani ZBa-alawi WKhodakarami FLin EYu YMartin SOrtmann JAittokallio THafner MHaibe-Kains B(2022)Evaluation of statistical approaches for association testing in noisy drug screening dataBMC Bioinformatics10.1186/s12859-022-04693-z23:1Online publication date: 18-May-2022
https://doi.org/10.1186/s12859-022-04693-z
Otero DParapar JBarreiro ÁHung CHong JBechini ASong E(2021)The wisdom of the rankersProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441947(672-680)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441947
Losada DHerrmann MElsweiler DHung CHong JBechini ASong E(2021)Cost-effective identification of on-topic search queries using multi-armed banditsProceedings of the 36th Annual ACM Symposium on Applied Computing10.1145/3412841.3441944(645-654)Online publication date: 22-Mar-2021
https://dl.acm.org/doi/10.1145/3412841.3441944
Cormack GGrossman MPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Quantifying Bias and Variance of System RankingsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331356(1089-1092)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331356
Kutlu MElsayed THasanain MLease MCuzzocrea AAllan JPaton NSrivastava DAgrawal RBroder AZaki MCandan SLabrinidis ASchuster AWang H(2018)When Rank Order Isn't EnoughProceedings of the 27th ACM International Conference on Information and Knowledge Management10.1145/3269206.3271751(397-406)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3269206.3271751
Losada DParapar JBarreiro A(2018)A rank fusion approach based on score distributions for prioritizing relevance assessments in information retrieval evaluationInformation Fusion10.1016/j.inffus.2017.04.00139:C(56-71)Online publication date: 1-Jan-2018
https://dl.acm.org/doi/10.1016/j.inffus.2017.04.001
Losada DParapar JBarreiro A(2018)When to stop making relevance judgments? A study of stopping methods for building information retrieval test collectionsJournal of the Association for Information Science and Technology10.1002/asi.2407770:1(49-60)Online publication date: 12-Dec-2018
https://dl.acm.org/doi/10.1002/asi.24077
Losada DParapar JBarreiro ÁOssowski S(2016)Feeling lucky?Proceedings of the 31st Annual ACM Symposium on Applied Computing10.1145/2851613.2851692(1027-1034)Online publication date: 4-Apr-2016
https://dl.acm.org/doi/10.1145/2851613.2851692
Baruah GRoegiest ASmucker MAllan JCroft Bde Vries AZhai C(2015)Pooling for User-Oriented Evaluation MeasuresProceedings of the 2015 International Conference on The Theory of Information Retrieval10.1145/2808194.2809493(341-344)Online publication date: 27-Sep-2015
https://dl.acm.org/doi/10.1145/2808194.2809493
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Validity and power of t-test for comparing MAP and GMAP

Hybrid pooling for enhancement of generalization ability in deep convolutional neural networks

Pooling semilattices and non-adaptive pooling designs