Article

Problems with Kendall's tau

Authors:

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

Pages 839 - 840

https://doi.org/10.1145/1277741.1277935

Published: 23 July 2007 Publication History

Get Access

Abstract

This poster describes a potential problem with a relatively well used measure in Information Retrieval research: Kendall's Tau rank correlation coefficient. The coefficient is best known for its use in determining the similarity of test collections when ranking sets of retrieval runs. Threshold values for the coefficient have been defined and used in a number of published studies in information retrieval. However, this poster presents results showing that basing decisions on such thresholds is not as reliableas has been assumed.

References

[1]

Bland, J.M., Altman, D.G. (1986) Statistical methods for assessing agreement between two methods of clinical measurement. Lancet, i, 307--310.

Crossref

Google Scholar

[2]

Buckley, C. & Voorhees, E.M. (2005) Retrieval evaluation with incomplete information, Proc. of ACM SIGIR, 25--32.

Digital Library

Google Scholar

[3]

Carterette, B. & Allan, J. (2005) Incremental test collections, Proc. of ACM SIGIR, 680--687.

Digital Library

Google Scholar

[4]

Kendall, M. (1938) A New Measure of Rank Correlation, Biometrika, 30, 81--89.

Crossref

Google Scholar

[5]

Lee, S., Myaeng, S.H., Kim, H., Seo, J.H., Lee, B., Cho, S. (2002) Characteristics of the Korean Test Collection for CLIR in NTCIR-3, Working Notes of NTCIR.

Google Scholar

[6]

Sanderson, M. & Joho, H. (2004) Forming test collections with no system pooling, Proc. of ACM SIGIR, 33--40.

Digital Library

Google Scholar

[7]

Voorhees, E.M. (1998) Variations in relevance judgments and the measurement of retrieval effectiveness, Proc. of ACM SIGIR, 315--323.

Digital Library

Google Scholar

[8]

Voorhees, E.M. (2001) Evaluation by highly relevant documents, Proc. of ACM SIGIR, 74--82.

Digital Library

Google Scholar

[9]

Yilmaz, E. & Aslam, J.A. (2006) Estimating Average Precision with Incomplete and Imperfect Judgments Proc. of ACM CIKM, 102--111.

Digital Library

Google Scholar

[10]

Yom-Tov, E, Fine, S., Carmel, D., Darlow, A. (2005) Learning to estimate query difficulty: including applications to missing content detection and distributed information retrieval, Proc. of ACM SIGIR, 512--519.

Digital Library

Google Scholar

Cited By

View all

Sheikh SVela BAttari VHuang XMorcos PHanagan JAcemi CKaraman IElwany AArroyave´ R(2024)Exploring chemistry and additive manufacturing design spaces: a perspective on computationally-guided design of printable alloysMaterials Research Letters10.1080/21663831.2024.231620412:4(235-263)Online publication date: 21-Feb-2024
https://doi.org/10.1080/21663831.2024.2316204
Vela BKhatamsaz DAcemi CKaraman IArróyave R(2023)Data-augmented modeling for yield strength of refractory high entropy alloys: A Bayesian approachActa Materialia10.1016/j.actamat.2023.119351261(119351)Online publication date: Dec-2023
https://doi.org/10.1016/j.actamat.2023.119351
Althammer SHofstätter SVerberne SHanbury AAl Hasan MXiong L(2022)TripJudgeProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557714(3801-3805)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557714
Show More Cited By

Index Terms

Problems with Kendall's tau
1. Information systems
  1. Information retrieval

Recommendations

On attainability of Kendall’s tau matrices and concordance signatures
Abstract
Methods are developed for checking and completing systems of bivariate and multivariate Kendall’s tau concordance measures in applications where only partial information about dependencies between variables is available. The concept of ...
A comparative analysis of Spearman's rho and Kendall's tau in normal and contaminated normal models

This paper analyzes the performances of Spearman's rho (SR) and Kendall's tau (KT) with respect to samples drawn from bivariate normal and contaminated normal populations. Theoretical and simulation results suggest that, contrary to the opinion of ...
Influence function-based confidence intervals for the Kendall rank correlation coefficient
Abstract
Correlation coefficients measure the association between two random variables. In circumstances in which the typically-used Pearson correlation coefficient does not suffice, the Kendall rank correlation coefficient is routinely used as an ...

Comments

Information & Contributors

Information

Published In

SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrieval

July 2007

946 pages

ISBN:9781595935977

DOI:10.1145/1277741

General Chairs:
Wessel Kraaij
TNO, The Netherlands
,
Arjen P. de Vries
CWI, The Netherlands
,
Program Chairs:
Charles L. A. Clarke
University of Waterloo, Canada
,
Norbert Fuhr
University of Duisburg-Essen, Germany
,
Noriko Kando
National Institute of Informatics, Japan

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tag

experimentation

Qualifiers

Article

Conference

SIGIR07

Sponsor:

SIGIR07: The 30th Annual International SIGIR Conference

July 23 - 27, 2007

Amsterdam, The Netherlands

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

19
Total Citations
View Citations
856
Total Downloads

Downloads (Last 12 months)20
Downloads (Last 6 weeks)5

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Sheikh SVela BAttari VHuang XMorcos PHanagan JAcemi CKaraman IElwany AArroyave´ R(2024)Exploring chemistry and additive manufacturing design spaces: a perspective on computationally-guided design of printable alloysMaterials Research Letters10.1080/21663831.2024.231620412:4(235-263)Online publication date: 21-Feb-2024
https://doi.org/10.1080/21663831.2024.2316204
Vela BKhatamsaz DAcemi CKaraman IArróyave R(2023)Data-augmented modeling for yield strength of refractory high entropy alloys: A Bayesian approachActa Materialia10.1016/j.actamat.2023.119351261(119351)Online publication date: Dec-2023
https://doi.org/10.1016/j.actamat.2023.119351
Althammer SHofstätter SVerberne SHanbury AAl Hasan MXiong L(2022)TripJudgeProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557714(3801-3805)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557714
Voorhees ERoberts KDiaz FShah CSuel TCastells PJones RSakai T(2021)On the Quality of the TREC-COVID IR Test CollectionsProceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3404835.3463244(2422-2428)Online publication date: 11-Jul-2021
https://dl.acm.org/doi/10.1145/3404835.3463244
Breuer TFerro NFuhr NMaistro MSakai TSchaer PSoboroff IHuang JChang YCheng XKamps JMurdock VWen JLiu Y(2020)How to Measure the Reproducibility of System-oriented IR ExperimentsProceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3397271.3401036(349-358)Online publication date: 25-Jul-2020
https://dl.acm.org/doi/10.1145/3397271.3401036
Cheba KBąk I(2020)Sustainable Development and Green Economy in the European Union Countries—Statistical AnalysisClassification and Data Analysis10.1007/978-3-030-52348-0_11(163-185)Online publication date: 29-Aug-2020
https://doi.org/10.1007/978-3-030-52348-0_11
Ziolo MFilipiak BBąk ICheba KTîrca DNovo-Corti I(2019)Finance, Sustainability and Negative Externalities. An Overview of the European ContextSustainability10.3390/su1115424911:15(4249)Online publication date: 6-Aug-2019
https://doi.org/10.3390/su11154249
Roitero KCulpepper JSanderson MScholer FMizzaro S(2019)Fewer topics? A million topics? Both?! On topics subsets in test collectionsInformation Retrieval Journal10.1007/s10791-019-09357-wOnline publication date: 8-May-2019
https://doi.org/10.1007/s10791-019-09357-w
Kutlu MElsayed THasanain MLease MCuzzocrea AAllan JPaton NSrivastava DAgrawal RBroder AZaki MCandan SLabrinidis ASchuster AWang H(2018)When Rank Order Isn't EnoughProceedings of the 27th ACM International Conference on Information and Knowledge Management10.1145/3269206.3271751(397-406)Online publication date: 17-Oct-2018
https://dl.acm.org/doi/10.1145/3269206.3271751
Magdy WElsayed THasanain M(2016)On the Evaluation of Tweet Timeline Generation TaskAdvances in Information Retrieval10.1007/978-3-319-30671-1_48(648-653)Online publication date: 2016
https://doi.org/10.1007/978-3-319-30671-1_48
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

On attainability of Kendall’s tau matrices and concordance signatures

A comparative analysis of Spearman's rho and Kendall's tau in normal and contaminated normal models

Influence function-based confidence intervals for the Kendall rank correlation coefficient