short-paper

Investigating the Reliability of Click Models

Authors:

Jiaxin Mao,

Zhumin Chu,

Yiqun Liu,

Min Zhang, and

Shaoping MaAuthors Info & Claims

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

September 2019

Pages 125 - 128

https://doi.org/10.1145/3341981.3344242

Published: 26 September 2019 Publication History

Get Access

Abstract

Click models aim to extract accurate relevance feedback from the noisy and biased user clicks. Previous work focuses on reducing the systematic bias between click and relevance but few studies have examined the reliability and precision of click models' relevance estimation. So in this study, we propose to investigate the reliability of relevance estimation derived by click models. Instead of getting a point estimate of relevance, a variational Bayesian method is used to infer the posterior distribution of relevance parameters. Based on the posterior distribution, we define measures for the reliability of pointwise and pairwise relevance estimation. With experiments on both real and synthetic query logs, we show that: 1) the proposed method effectively captures the uncertainty in relevance estimation; 2) the reliability of click models' relevance estimation is affected by the size of training data, the average ranking position of documents, and the ranking strategy of search engines.

References

[1]

Matthew James Beal et almbox. 2003. Variational algorithms for approximate Bayesian inference .university of London London.

Google Scholar

[2]

Alexey Borisov, Ilya Markov, Maarten de Rijke, and Pavel Serdyukov. 2016. A neural click model for web search. In Proceedings of the 25th International Conference on World Wide Web. International World Wide Web Conferences Steering Committee, 531--541.

Digital Library

Google Scholar

[3]

Olivier Chapelle and Ya Zhang. 2009. A dynamic bayesian network click model for web search ranking. In WWW '09. ACM, 1--10.

Digital Library

Google Scholar

[4]

Aleksandr Chuklin, Ilya Markov, and Maarten de Rijke. 2015. Click models for web search. Synthesis Lectures on Information Concepts, Retrieval, and Services, Vol. 7, 3 (2015), 1--115.

Crossref

Google Scholar

[5]

Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008. An experimental comparison of click position-bias models. In WSDM'08. ACM, 87--94.

Digital Library

Google Scholar

[6]

Georges E Dupret and Benjamin Piwowarski. 2008. A user browsing model to predict search engine click data from past observations. In Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval. ACM, 331--338.

Digital Library

Google Scholar

[7]

Arjun K Gupta and Saralees Nadarajah. 2004. Handbook of beta distribution and its applications .CRC press.

Google Scholar

[8]

Thorsten Joachims, Laura Granka, Bing Pan, Helene Hembrooke, and Geri Gay. 2005. Accurately interpreting clickthrough data as implicit feedback. In SIGIR'05. Acm, 154--161.

Digital Library

Google Scholar

[9]

Xiangsheng Li, Jiaxin Mao, Chao Wang, Yiqun Liu, Min Zhang, and Shaoping Ma. 2019. Teach Machine How to Read: Reading Behavior Inspired Relevance Estimation. In SIGIR'19 .

Digital Library

Google Scholar

[10]

Chao Liu, Fan Guo, and Christos Faloutsos. 2009. Bbm: bayesian browsing model from petabyte-scale data. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining. ACM, 537--546.

Digital Library

Google Scholar

[11]

Cheng Luo, Yukun Zheng, Jiaxin Mao, Yiqun Liu, Min Zhang, and Shaoping Ma. 2017. Training deep ranking model with weak relevance labels. In Australasian Database Conference . Springer, 205--216.

Crossref

Google Scholar

[12]

Jiaxin Mao, Cheng Luo, Min Zhang, and Shaoping Ma. 2018. Constructing Click Models for Mobile Search. In The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval. ACM, 775--784.

Google Scholar

[13]

Chao Wang, Yiqun Liu, Min Zhang, Shaoping Ma, Meihong Zheng, Jing Qian, and Kuo Zhang. 2013. Incorporating vertical results into search click models. In SIGIR'13. ACM, 503--512.

Digital Library

Google Scholar

Cited By

View all

Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Zou LHao CCai HWang SCheng SCheng ZYe WGu SYin DAl Hasan MXiong L(2022)Approximated Doubly Robust Search Relevance EstimationProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557145(3756-3765)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557145
Fairstein YHaramaty ELazerson ALewin-Eytan LSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)External Evaluation of Ranking Models under Extreme Position-BiasProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498420(252-261)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498420
Show More Cited By

Index Terms

Investigating the Reliability of Click Models
1. Information systems
  1. Information retrieval
    1. Users and interactive retrieval
  2. World Wide Web
    1. Web searching and information discovery
      1. Web search engines
2. Theory of computation
  1. Theory and algorithms for application domains
    1. Machine learning theory
      1. Bayesian analysis

Recommendations

Constructing Click Models for Mobile Search
SIGIR '18: The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval

Users' click-through behavior is considered as a valuable yet noisy source of implicit relevance feedback for web search engines. A series of click models have therefore been proposed to extract accurate and unbiased relevance feedback from click logs. ...
Read More
A Context-Aware Click Model for Web Search
WSDM '20: Proceedings of the 13th International Conference on Web Search and Data Mining

To better exploit the search logs, various click models have been proposed to extract implicit relevance feedback from user clicks. Most traditional click models are based on probability graphical models (PGMs) with manually designed dependencies. ...
Read More
Characterizing search intent diversity into click models
WWW '11: Proceedings of the 20th international conference on World wide web

Modeling a user's click-through behavior in click logs is a challenging task due to the well-known position bias problem. Recent advances in click models have adopted the examination hypothesis which distinguishes document relevance from position bias. ...
Read More

Comments

Information & Contributors

Information

Published In

ICTIR '19: Proceedings of the 2019 ACM SIGIR International Conference on Theory of Information Retrieval

September 2019

273 pages

ISBN:9781450368810

DOI:10.1145/3341981

General Chairs:
Yi Fang
Santa Clara University, USA
,
Yi Zhang
University of California, Santa Cruz, USA
,
Program Chairs:
James Allan
University of Massachusetts, Amherst, USA
,
Krisztian Balog
University of Stavanger, Norway
,
Ben Carterette
Spotify, USA
,
Jiafeng Guo
Chinese Academy of Sciences, China

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 September 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Short-paper

Funding Sources

National Key Research and Development Program of China
Natural Science Foundation of China

Conference

ICTIR '19

Sponsor:

SIGIR

ICTIR '19: The 2019 ACM SIGIR International Conference on the Theory of Information Retrieval

October 2 - 5, 2019

CA, Santa Clara, USA

Acceptance Rates

ICTIR '19 Paper Acceptance Rate 20 of 41 submissions, 49%;

Overall Acceptance Rate 209 of 482 submissions, 43%

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
141
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

View all

Breuer TFuhr NSchaer P(2023)Validating Synthetic Usage Data in Living Lab EnvironmentsJournal of Data and Information Quality10.1145/3623640Online publication date: 24-Sep-2023
https://dl.acm.org/doi/10.1145/3623640
Zou LHao CCai HWang SCheng SCheng ZYe WGu SYin DAl Hasan MXiong L(2022)Approximated Doubly Robust Search Relevance EstimationProceedings of the 31st ACM International Conference on Information & Knowledge Management10.1145/3511808.3557145(3756-3765)Online publication date: 17-Oct-2022
https://dl.acm.org/doi/10.1145/3511808.3557145
Fairstein YHaramaty ELazerson ALewin-Eytan LSelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)External Evaluation of Ranking Models under Extreme Position-BiasProceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498420(252-261)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498420
Dai XXi YZhang WLiu QTang RHe XHou JWang JYu Y(2021)Beyond Relevance Ranking: A General Graph Matching Framework for Utility-Oriented Learning to RankACM Transactions on Information Systems10.1145/346430340:2(1-29)Online publication date: 16-Nov-2021
https://dl.acm.org/doi/10.1145/3464303
Ai QYang TWang HMao J(2021)Unbiased Learning to RankACM Transactions on Information Systems10.1145/343986139:2(1-29)Online publication date: 17-Feb-2021
https://dl.acm.org/doi/10.1145/3439861

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Constructing Click Models for Mobile Search

A Context-Aware Click Model for Web Search

Characterizing search intent diversity into click models