Intervention harvesting for context-dependent examination-bias estimation

Z Fang, A Agarwal, T Joachims - … of the 42nd international ACM SIGIR …, 2019 - dl.acm.org
Z Fang, A Agarwal, T Joachims
Proceedings of the 42nd international ACM SIGIR conference on research and …, 2019dl.acm.org
Accurate estimates of examination bias are crucial for unbiased learning-to-rank from
implicit feedback in search engines and recommender systems, since they enable the use of
Inverse Propensity Score (IPS) weighting techniques to address selection biases and
missing data. Unfortunately, existing examination-bias estimators are limited to the Position-
Based Model (PBM), where the examination bias may only depend on the rank of the
document. To overcome this limitation, we propose a Contextual Position-Based Model …
Accurate estimates of examination bias are crucial for unbiased learning-to-rank from implicit feedback in search engines and recommender systems, since they enable the use of Inverse Propensity Score (IPS) weighting techniques to address selection biases and missing data. Unfortunately, existing examination-bias estimators are limited to the Position-Based Model (PBM), where the examination bias may only depend on the rank of the document. To overcome this limitation, we propose a Contextual Position-Based Model (CPBM) where the examination bias may also depend on a context vector describing the query and the user. Furthermore, we propose an effective estimator for the CPBM based on intervention harvesting. A key feature of the estimator is that it does not require disruptive interventions but merely exploits natural variation resulting from the use of multiple historic ranking functions. Real-world experiments on the ArXiv search engine and semi-synthetic experiments on the Yahoo Learning-To-Rank dataset demonstrate the superior effectiveness and robustness of the new approach.
ACM Digital Library