Cited By
View all- Xie Y(2021)Research on Vertical Search Method of Multidimensional Resources in English Discipline Based on Edge ComputingMobile Information Systems10.1155/2021/55181352021Online publication date: 1-Jan-2021
Vertical Search Blending: A Real-world Counterfactual Dataset
Authors: 
Pavel Procházka, Matej Kocián, Jakub Drdák, Jan Vršovský, Vladimír Kadlec, Jaroslav KucharAuthors Info & Claims
Pavel Procházka, Matej Kocián, Jakub Drdák, Jan Vršovský, Vladimír Kadlec, Jaroslav KucharAuthors Info & ClaimsPages 1237 - 1240
Abstract
Blending of search results from several vertical sources became standard among web search engines. Similar scenarios appear in computational advertising, news recommendation, and other interactive systems. As such environments give only partial feedback, the evaluation of new policies conventionally requires expensive online A/B tests. Counterfactual approach is a promising alternative, nevertheless, it requires specific conditions for a valid off-policy evaluation. We release a large-scale, real-world vertical-blending dataset gathered bySeznam.cz web search engine. The dataset contains logged partial feedback with the corresponding propensity and is thus suited for counterfactual evaluation. We provide basic checks for validity and evaluate several learning methods.
References
[1]
Jaime Arguello. 2017. Aggregated Search. Found. Trends Inf. Retr. 10, 5 (March 2017), 365--502.
[2]
Jie at al. 2013. A Unified Search Federation System Based on Online User Feedback. In Proceedings of the 19th ACM SIGKDD (KDD '13). ACM, New York, NY, USA, 1195--1203.
[3]
Vorobev at al. 2015. Gathering Additional Feedback on Search Results by Multi-Armed Bandits with Respect to Production Ranking. In Proceedings of the 24th International Conference on WWW (WWW '15). International World Wide Web Conferences Steering Committee, Republic and Canton of Geneva, Switzerland, 1177--1187.
[4]
Miroslav Dudík, John Langford, and Lihong Li. 2011. Doubly Robust Policy Evaluation and Learning. In Proceedings of the 28th ICML (ICML'11). Omnipress,USA, 1097--1104.
[5]
Alexandre Gilotte, Clément Calauzènes, Thomas Nedelec, Alexandre Abraham,and Simon Dollé. 2018. Offline A/B Testing for Recommender Systems. In Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining (WSDM '18). ACM, New York, NY, USA, 198--206.
[6]
Katja Hofmann, Shimon Whiteson, and Maarten de Rijke. 2011. Contextual Bandits for Information Retrieval. In NIPS 2011: Workshop on Bayesian Optimization,Experimental Design and Bandits: Theory and Applications, Vol. 12.
[7]
Thorsten Joachims and Adith Swaminathan. 2016. SIGIR Tutorial on Counterfactual Evaluation and Learning for Search, Recommendation and Ad Placement. In Proceedings of the 39th International ACM SIGIR. ACM, 1199--1201.
[8]
Thorsten Joachims, Adith Swaminathan, and Maarten de Rijke. 2018. Deep Learning with Logged Bandit Feedback. In International Conference on Learning Representations(iclr 2018 ed.).
[9]
Oren Kurland and J. Shane Culpepper. 2018. Fusion in Information Retrieval:SIGIR 2018 Half-Day Tutorial. In The 41st International ACM SIGIR (SIGIR '18). ACM, New York, NY, USA, 1383--1386.
[10]
John Langford, Lihong Li, and Alexander Strehl. 2007. Vowpal wabbit opensource project. (2007). https://github.com/VowpalWabbit/vowpalwabbit/wiki
[11]
Damien Lefortier, Adith Swaminathan, Xiaotao Gu, Thorsten Joachims, and Maarten de Rijke. 2016. Large-scale Validation of Counterfactual Learning Methods: A Test-Bed. CoRRabs/1612.00367 (2016). arXiv:1612.00367
[12]
Lihong Li, Shunbao Chen, Jim Kleban, and Ankur Gupta. 2015. Counterfactual estimation and optimization of click metrics in search engines: A case study. In Proceedings of the 24th International Conference on WWW. ACM, 929--934.
[13]
Lihong Li, Wei Chu, John Langford, and Robert E Schapire. 2010. A contextual-bandit approach to personalized news article recommendation. In Proceedings of the 19th international conference on WWW. ACM, 661--670.
[14]
James McInerney, Benjamin Lacker, Samantha Hansen, Karl Higley, Hugues Bouchard, Alois Gruson, and Rishabh Mehrotra. 2018. Explore, Exploit, and Ex-plain: Personalizing Explainable Recommendations with Bandits. In Proceedings of the 12th ACM Conference on Recommender Systems (RecSys '18). ACM, New York, NY, USA, 31--39.
[15]
Paul R. Rosenbaum and Donald B. Rubin. 1983. The central role of the propensity score in observational studies for causal effects. Biometrika 70, 1 (04 1983), 41--55.
[16]
Daniel J. Russo, Benjamin Van Roy, Abbas Kazerouni, Ian Osband, and Zheng Wen. 2018. A Tutorial on Thompson Sampling. Foundations and Trends in Machine Learning 11, 1 (2018), 1--96.
[17]
Marc Sloan and Jun Wang. 2012. Dynamical Information Retrieval Modelling: A Portfolio-armed Bandit Machine Approach. In Proceedings of the 21st International Conference on WWW (WWW '12 Companion). ACM, New York, NY, USA, 603--604.
[18]
Adith Swaminathan and Thorsten Joachims. 2015. The self-normalized estimator for counterfactual learning. In Advances in Neural Information Processing Systems.3231--3239.
[19]
Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miroslav Dudík,John Langford, Damien Jose, and Imed Zitouni. 2016. Off-policy evaluation for slate recommendation. CoRRabs/1605.04812 (2016). arXiv:1605.04812 http://arxiv.org/abs/1605.04812
Index Terms
- Vertical Search Blending: A Real-world Counterfactual Dataset
Recommendations
Focused ranking in a vertical search engine
SIGIR '07: Proceedings of the 30th annual international ACM SIGIR conference on Research and development in information retrievalSince the debut of PageRank and HITS, hyperlink-induced Web document ranking has come a long way. The Web has become increasingly vast and topically diverse. Such vastness has led many into the area of topic-sensitive ranking and its variants. We ...
Smoothing Click Counts for Aggregated Vertical Search
ECIR 2011: Proceedings of the 33rd European Conference on Advances in Information Retrieval - Volume 6611Clickthrough data is a critical feature for improving web search ranking. Recently, many search portals have provided aggregated search, which retrieves relevant information from various heterogeneous collections called verticals. In addition to the ...
Comments
Information & Contributors
Information
Published In
July 2019
1512 pages
ISBN:9781450361729
DOI:10.1145/3331184
- General Chairs:
- Benjamin Piwowarski,
- Max Chevalier,
- Eric Gaussier,
- Program Chairs:
- Yoelle Maarek,
- Jian-Yun Nie,
- Falk Scholer
Copyright © 2019 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 18 July 2019
Check for updates
Author Tags
Qualifiers
- Short-paper
Conference
SIGIR '19
Sponsor:
SIGIR '19: The 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval
July 21 - 25, 2019
Paris, France
Acceptance Rates
SIGIR'19 Paper Acceptance Rate 84 of 426 submissions, 20%;
Overall Acceptance Rate 792 of 3,983 submissions, 20%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- View Citations1Total Citations
- 360Total Downloads
- Downloads (Last 12 months)4
- Downloads (Last 6 weeks)1
Reflects downloads up to 20 Feb 2025
Other Metrics
Citations
Cited By
View all- Xie Y(2021)Research on Vertical Search Method of Multidimensional Resources in English Discipline Based on Edge ComputingMobile Information Systems10.1155/2021/55181352021Online publication date: 1-Jan-2021
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in