research-article

Proof by experimentation?: towards better IR research

Author:

Norbert FuhrAuthors Info & Claims

ACM SIGIR Forum, Volume 54, Issue 2

Article No.: 2, Pages 1 - 4

https://doi.org/10.1145/3483382.3483385

Published: 20 August 2021 Publication History

Get Access

Abstract

Most IR experiments lack both internal and external validity. Top performance is mostly an illusion, given the lack of solid statistical evidence. Thus, reviewers should ignore performance numbers in their judgement, and pay more attention to proper methodology. Researchers need to focus more on performance analysis and prediction.

References

[1]

Norbert Fuhr. Some common mistakes in IR evaluation, and how they can be avoided. SIGIR Forum, 51(3):32--41, 2017. URL http://sigir.org/wp-content/uploads/2018/01/p032.pdf.

Digital Library

Google Scholar

[2]

Marco Ferrante, Nicola Ferro, and Silvia Pontarollo. Are IR evaluation measures on an interval scale? In Proc. ICTIR, page 67--74, 2017.

Digital Library

Google Scholar

[3]

Marco Ferrante, Nicola Ferro, and Norbert Fuhr. Towards meaningful statements in IR evaluation. mapping evaluation measures to interval scales. Submitted for publication, 2021.

Google Scholar

[4]

Ben Carterette. The best published result is random: Sequential testing and its effect on reported effectiveness. In Proc. SIGIR, pages 747--750, 2015.

Digital Library

Google Scholar

[5]

Nicola Ferro et al. The Dagstuhl Perspectives Workshop on performance modeling and prediction. SIGIR Forum, 52(1):91--101, 2018.

Digital Library

Google Scholar

Cited By

View all

Krause TDeriyeva ABeinke JBartels GThomas O(2024)Mitigating Exposure Bias in Recommender Systems – A Comparative Analysis of Discrete Choice ModelsACM Transactions on Recommender Systems10.1145/3641291Online publication date: 27-Jan-2024
https://dl.acm.org/doi/10.1145/3641291
Petrov AMacdonald C(2023)RSS: Effective and Efficient Training for Sequential Recommendation using Recency SamplingACM Transactions on Recommender Systems10.1145/3604436Online publication date: 23-Jun-2023
https://doi.org/10.1145/3604436
Fröbe MReimer JMacAvaney SDeckers NReich SBevendorff JStein BHagen MPotthast MChen HDuh WHuang HKato MMothe JPoblete B(2023)The Information Retrieval Experiment PlatformProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591888(2826-2836)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591888
Show More Cited By

Index Terms

Proof by experimentation?: towards better IR research
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results

Index terms have been assigned to the content through auto-classification.

Recommendations

Proof by Experimentation? Towards Better IR Research
SIGIR '20: Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval

The current fight against the COVID-19 pandemic illustrates the importance of proper scientific methods: Besides fake news lacking any factual evidence, reports on clinical trials with various drugs often yield contradicting results; here, only a closer ...
Read More
Supporting Interface Experimentation for Blockchain Applications
NordiCHI '22: Adjunct Proceedings of the 2022 Nordic Human-Computer Interaction Conference

There is an increasingly diverse range of smart-contract blockchains on which decentralized applications (dApps) are built. However, HCI research has so far failed to address them, focusing primarily on Bitcoin and Ethereum. This is problematic as these ...
Read More
Proof-of-stake at stake: predatory, destructive attack on PoS cryptocurrencies
CryBlock '20: Proceedings of the 3rd Workshop on Cryptocurrencies and Blockchains for Distributed Systems

There have been several 51% attacks on Proof-of-Work (PoW) blockchains recently, including Verge and GameCredits, but the most noteworthy has been the attack that saw hackers make off with up to $18 million after a successful double-spend was executed ...
Read More

Comments

Information & Contributors

Information

Published In

ACM SIGIR Forum Volume 54, Issue 2

December 2020

115 pages

ISSN:0163-5840

DOI:10.1145/3483382

Issue’s Table of Contents

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 20 August 2021

Published in SIGIR Volume 54, Issue 2

Check for updates

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
50
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

View all

Krause TDeriyeva ABeinke JBartels GThomas O(2024)Mitigating Exposure Bias in Recommender Systems – A Comparative Analysis of Discrete Choice ModelsACM Transactions on Recommender Systems10.1145/3641291Online publication date: 27-Jan-2024
https://dl.acm.org/doi/10.1145/3641291
Petrov AMacdonald C(2023)RSS: Effective and Efficient Training for Sequential Recommendation using Recency SamplingACM Transactions on Recommender Systems10.1145/3604436Online publication date: 23-Jun-2023
https://doi.org/10.1145/3604436
Fröbe MReimer JMacAvaney SDeckers NReich SBevendorff JStein BHagen MPotthast MChen HDuh WHuang HKato MMothe JPoblete B(2023)The Information Retrieval Experiment PlatformProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3591888(2826-2836)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3591888
Breuer TKeller JSchaer PAmigo ECastells PGonzalo JCarterette BCulpepper JKazai G(2022)ir_metadata: An Extensible Metadata Schema for IR ExperimentsProceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3477495.3531738(3078-3089)Online publication date: 6-Jul-2022
https://dl.acm.org/doi/10.1145/3477495.3531738

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations