article

Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search

Authors:

Thorsten Joachims,

Helene Hembrooke,

Filip Radlinski, and

Geri GayAuthors Info & Claims

ACM Transactions on Information Systems (TOIS), Volume 25, Issue 2

Pages 7 - es

https://doi.org/10.1145/1229179.1229181

Published: 01 April 2007 Publication History

Abstract

This article examines the reliability of implicit feedback generated from clickthrough data and query reformulations in World Wide Web (WWW) search. Analyzing the users' decision process using eyetracking and comparing implicit feedback against manual relevance judgments, we conclude that clicks are informative but biased. While this makes the interpretation of clicks as absolute relevance judgments difficult, we show that relative preferences derived from clicks are reasonably accurate on average. We find that such relative preferences are accurate not only between results from an individual query, but across multiple sets of results within chains of query reformulations.

References

[1]

Agichtein, E., Brill, E., and Dumais, S. 2006a. Improving Web search ranking by incorporating user behavior. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR). 19--26.

[2]

Agichtein, E., Brill, E., Dumais, S., and Ragno, R. 2006b. Learning user interaction models for predicting Web search preferences. In Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR). 3--10.

[3]

Almeida, R. and Almeida, V. 2004. A community-aware search engine. In Proceedings of the World Wide Web Conference (WWW).

[4]

Bartell, B., Cottrell, G., and Belew, R. 1994. Automatic combination of multiple ranked retrieval systems. In Proceedings of the Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). 173--181.

[5]

Belew, R. 2000. Finding Out About. Cambridge, University Press, Cambridge, U.K.

[6]

Boyan, J., Freitag, D., and Joachims, T. 1996. A machine learning architecture for optimizing Web search engines. In Proceedings of the AAAI Workshop on Internet Based Information Systems. 1--8.

[7]

Broder, A. 2002. A taxonomy of web search. ACM SIGIR For. 36, 2, 3--10.

[8]

Brumby, D. and Howes, A. 2003. Interdependence and past-experience in menu choice assessment. In Poster presented at the 25th Annual Meeting of the Cognitive Science Society.

[9]

Brumby, D. and Howes, A. 2004. Good enough but I'll just check: Web-page search as attentional refocusing. In Proceedings of the International Conference on Cognitive Modeling.

[10]

Burges, C., Renshaw, S., Renshaw, E., Ari, L., Deeds, M., Hamilton, N., and Hullender, G. 2005. Learning to rank using gradient descent. In Proceedings of the International Conference on Machine Learning (ICML, Bonn, Germany).

[11]

Claypool, M., Le, P., Waseda, M., and Brown, D. 2001. Implicit interest indicators. In Proceedings of the International Conference on Intelligent User Interfaces (IUI). 33--40.

[12]

Cohen, W., Shapire, R., and Singer, Y. 1999. Learning to order things. J. Artific. Intell. Res. 10, 243--270.

[13]

Crammer, K. and Singer, Y. 2001. Pranking with ranking. In Proceedings of the Conference on Advances in Neural Information Processing Systems (NIPS).

[14]

Fox, S., Karnawat, K., Mydland, M., Dumais, S., and White, T. 2005. Evaluating implicit measures to improve web search. ACM Trans. Inform. Syst. 23, 2, 147--168.

[15]

Freund, Y., Iyer, R., Schapire, R., and Singer, Y. 1998. An efficient boosting algorithm for combining preferences. In International Conference on Machine Learning (ICML). Morgan Kaufmann San Francisco, CA, 170--178.

[16]

Fuhr, N. 1989. Optimum polynomial retrieval functions based on the probability ranking principle. ACM Trans. Inform. Syst. 7, 3, 183--204.

[17]

Furnas, G. 1985. Experience with an adaptive indexing scheme. In Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI). ACM Press, New York, NY, 131--135.

[18]

Goldberg, J., Stimson, M., Lewenstein, M., Scott, M., and Wichansky, A. 2002. Eye-tracking in Web search tasks: design implications. In Proceedings of the Eye tracking Research and Applications Symposium (ETRA). 51--58.

[19]

Granka, L., Joachims, T., and Gay, G. 2004. Eye-tracking analysis of user behavior in www search. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR).

[20]

Halverson, T. and Hornof, A. 2004. Link colors guide a search. In Proceedings of the ACM Conference on Computer-Human Interaction (CHI).

[21]

Herbrich, R., Graepel, T., and Obermayer, K. 2000. Large margin rank boundaries for ordinal regression. In Advances in Large Margin Classifiers. MIT Press, Cambridge, MA, 115--132.

[22]

Holland, S., Ester, M., and Kieling, W. 2003. Preference mining: A novel approach on mining user preferences for personalized applications. In Proceedings of the Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD). 204--216.

[23]

Hutchinson, W. and Alba, J. 1991. Ignoring irrelevant information: Situational determinats of consumer learning. J. Consum. Res. 18, 325--345.

[24]

Jansen, B., Spink, A., and Saracevic, T. 2000. Real life, real users, and real needs: A study and analysis of user queries on the web. Inf. Process. Manage. 36, 2, 207--227.

[25]

Joachims, T. 2002. Optimizing search engines using clickthrough data. In Proceedings of the ACM Conference on Knowledge Discovery and Data Mining (KDD).

[26]

Joachims, T., Freitag, D., and Mitchell, T. 1997. WebWatcher: A tour guide for the World Wide Web. In Proceedings of the International Joint Conference on Artificial Intelligence (IJCAI). Vol. 1. Morgan Kaufmann, San Francisco, CA, 770--777.

[27]

Joachims, T., Granka, L., Pang, B., Hembrooke, H., and Gay, G. 2005. Accurately interpreting clickthrough data as implicit feedback. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 154--161.

[28]

Jones, R. and Fain, D. 2003. Query word deletion prediction. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM Press, New York, NY, 435--436.

[29]

Just, M. and Carpenter, P. 1980. A theory of reading: From eye fixations to comprehension. Psycholog. Rev. 87, 329--354.

[30]

Kelly, D. and Belkin, N. 2004. Display time as implicit feedback: Understanding task effects. In Proceedings of the ACM International Conference on Research and Development in Information Retrieval (SIGIR). 377--384.

[31]

Kelly, D. and Teevan, J. 2003. Implicit feedback for inferring user preference: A bibliography. ACM SIGIR For. 37, 2, 18--28.

[32]

Kemp, D. and Ramamohanarao, K. 2002. Long-term learning for Web search engines. In Proceedings of the European Conference on Principles and Practice of Knowledge Discovery in Databases (PKDD). 263--274.

[33]

Klöckner, K., Wirschum, N., and Jameson, A. 2004. Depth- and breadth-first processing of search result lists. In Proceedings of the ACM Conference on Computer-Human Interaction. (Extended abstract.)

[34]

Lankford, C. 2000. Gazetracker: Software designed to facilitate eye movement analysis. In Proceedings of the Conference on Eye Tracking Research & Applications. 51--55.

[35]

Lieberman, H. 1995. Letizia: An agent that assists Web browsing. In Proceedings of the Fifteenth International Joint Conference on Artificial Intelligence (IJCAI '95). Morgan Kaufmann, San Francisco, CA, 924--929.

[36]

Mantell, S. and Kardes, F. 1999. The role of direction of comparison, attribute-based processing, and attitude-based processing in consumer preference. J. Consum. Res. 25, 335--352.

[37]

Morita, M. and Shinoda, Y. 1994. Information filtering based on user behavior analysis and best match text retrieval. In Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR). 272--281.

[38]

Oard, D. and Kim, J. 1998. Implicit feedback for recommender systems. In Proceedings of the AAAI Workshop on Recommender Systems. 81--83.

[39]

Pan, B., Hembrooke, H., Gay, G., Granka, L., Feusner, M., and Newman, J. 2004. The determinants of Web page viewing behavior: An eye tracking study. In Proceedings of the Conference on Eye Tracking Research & Applications, S. Spencer, Ed. ACM Press, New York, NY.

[40]

Radlinski, F. and Joachims, T. 2005. Query chains: Learning to rank from implicit feedback. In Proceedings of the ACM SIGKDD International Conference On Knowledge Discovery and Data Mining (KDD).

[41]

Rayner, K. 1998. Eye movements in reading and information processing. Psycholog. Bull. 124, 372--252.

[42]

Salogarvi, J., Kojo, I., Jaana, S., and Kaski, S. 2003. Can relevance be inferred from eye movements in information retrieval&quest; In Proceedings of the Workshop on Self-Organizing Maps. 261--266.

[43]

Samuelson, P. 1948. Consumption theory in terms of revealed preferences. Econometrica 15, 243--253.

[44]

Shen, X., Tan, B., and Zhai, C. 2005. Context-sensitive information retrieval using implicit feedback. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM Press, New York, NY, 43--50.

[45]

Silverstein, C., Marais, H., Henzinger, M., and Moricz, M. 1999. Analysis of a very large web search engine query log. ACM SIGIR Forum 33, 1, 6--12.

[46]

Teevan, J., Dumais, S., and Horvitz, E. 2005. Beyond the commons: Investigating the value of personalizing Web search. In Proceedings of the Workshop on New Technologies for Personalized Information Access (PIA). 84--92.

[47]

Varian, H. 1992. Microeconomic Analysis. Noron, New York, NY.

[48]

White, R., Ruthven, I., and Jose, J. 2002. The use of implicit evidence for relevance feedback in Web retrieval. In Proceedings of the 24th BCS-IRSG European Colloquium on IR Research. Springer-Verlag, London, U.K., 93--109.

[49]

White, R., Ruthven, I., and Jose, J. 2005. A study of factors affecting the utility of implicit relevance feedback. In Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR). ACM Press, New York, NY, 35--42.

Cited By

He LZhao JGu YElbaz MDing Z(2024)A bias study and an unbiased deep neural network for recommender systemsWeb Intelligence10.3233/WEB-23003622:1(15-29)Online publication date: 26-Mar-2024
https://doi.org/10.3233/WEB-230036
Gausen AMitra BLindley S(2024)A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to WorkersProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658900(207-220)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658900
Bink MElsweiler D(2024)Balancing Act: Boosting Strategies for Informed Search on Controversial TopicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638329(254-265)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638329
Show More Cited By

Index Terms

Evaluating the accuracy of implicit feedback from clicks and query reformulations in Web search
1. Information systems
  1. Information retrieval

Recommendations

Query chains: learning to rank from implicit feedback
KDD '05: Proceedings of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining

This paper presents a novel approach for using clickthrough data to learn ranked retrieval functions for web search results. We observe that users searching the web often perform a sequence, or chain, of queries with a similar information need. Using ...
Read More
Click data as implicit relevance feedback in web search

Search sessions consist of a person presenting a query to a search engine, followed by that person examining the search results, selecting some of those search results for further review, possibly following some series of hyperlinks, and perhaps ...
Read More
Impact of query intent and search context on clickthrough behavior in sponsored search

Implicit feedback techniques may be used for query intent detection, taking advantage of user behavior to understand their interests and preferences. In sponsored search, a primary concern is the user's interest in purchasing or utilizing a commercial ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Information Systems

ACM Transactions on Information Systems Volume 25, Issue 2

April 2007

141 pages

ISSN:1046-8188

EISSN:1558-2868

DOI:10.1145/1229179

Issue’s Table of Contents

Copyright © 2007 ACM.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 April 2007

Published in TOIS Volume 25, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

462
Total Citations
View Citations
3,149
Total Downloads

Downloads (Last 12 months)55
Downloads (Last 6 weeks)1

Other Metrics

View Author Metrics

Citations

Cited By

He LZhao JGu YElbaz MDing Z(2024)A bias study and an unbiased deep neural network for recommender systemsWeb Intelligence10.3233/WEB-23003622:1(15-29)Online publication date: 26-Mar-2024
https://doi.org/10.3233/WEB-230036
Gausen AMitra BLindley S(2024)A Framework for Exploring the Consequences of AI-Mediated Enterprise Knowledge Access and Identifying Risks to WorkersProceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency10.1145/3630106.3658900(207-220)Online publication date: 3-Jun-2024
https://dl.acm.org/doi/10.1145/3630106.3658900
Bink MElsweiler D(2024)Balancing Act: Boosting Strategies for Informed Search on Controversial TopicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638329(254-265)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638329
Buchholz ALondon BDi Benedetto GLichtenberg JStein YJoachims THui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Counterfactual Ranking Evaluation with Flexible Click ModelsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657810(1200-1210)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657810
Luo DZou LAi QChen ZLi CYin DDavison BHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Unbiased Learning-to-Rank Needs Unconfounded Propensity EstimationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657772(1535-1545)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657772
Zhang PYan YZhang XLi CWang SHuang FKim SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)TransGNN: Harnessing the Collaborative Power of Transformers and Graph Neural Networks for Recommender SystemsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657721(1285-1295)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657721
Balog KZhai CChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Tutorial on User Simulation for Evaluating Information Access Systems on the WebCompanion Proceedings of the ACM on Web Conference 202410.1145/3589335.3641243(1254-1257)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641243
Mao HZou LZheng YTang JChu XZhao JWang QYin DChua TNgo CKa-Wei Lee RKumar RLauw H(2024)Whole Page Unbiased Learning to RankProceedings of the ACM on Web Conference 202410.1145/3589334.3645474(1431-1440)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589334.3645474
Wang ZSun HDing CZhang X(2024)Does public environmental concern cause pollution transfer? Evidence from Chinese firms' off-site investmentsJournal of Cleaner Production10.1016/j.jclepro.2024.142825466(142825)Online publication date: Aug-2024
https://doi.org/10.1016/j.jclepro.2024.142825
Epstein RLothringer MZankich V(2024)How a Daily Regimen of Operant Conditioning Might Explain the Power of the Search Engine Manipulation Effect (SEME)Behavior and Social Issues10.1007/s42822-023-00155-0Online publication date: 5-Jan-2024
https://doi.org/10.1007/s42822-023-00155-0
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents