Proof by Experimentation? Towards Better IR Research

The current fight against the COVID-19 pandemic illustrates the importance of proper scientific methods: Besides fake news lacking any factual evidence, reports on clinical trials with various drugs often yield contradicting results; here, only a closer look at the underlying empirical methodology can help in forming a clearer picture.

In IR research, empirical foundation in the form of experiments plays an important role. However, the methods applied often are not at the level of scientific standards that hold in many other disciplines, as IR experiments are frequently flawed in several ways: Measures like MRR or ERR are invalid by definition, and MAP is based on unrealistic assumptions about user behaviour; computing relative improvements of arithmetic means is statistical nonsense; test hypotheses often are formulated after the experiment has been carried out; multiple hypotheses are tested without correction; many experiments are not reproducible results or are compared to weak baselines [1, 6]; frequent reuse of the same test collections yields random results [2]; authors (and reviewers) believe that experiments prove the claims made. Methods for overcoming these problems have been pointed out [5], but are still widely ignored.