Article

Using clicks as implicit judgments: expectations versus observations

Authors:

Milad Shokouhi,

Bodo Billerbeck,

Andrew TurpinAuthors Info & Claims

ECIR'08: Proceedings of the IR research, 30th European conference on Advances in information retrieval

Pages 28 - 39

Published: 30 March 2008 Publication History

Abstract

Clickthrough data has been the subject of increasing popularity as an implicit indicator of user feedback. Previous analysis has suggested that user click behaviour is subject to a quality bias--that is, users click at different rank positions when viewing effective search results than when viewing less effective search results. Based on this observation, it should be possible to use click data to infer the quality of the underlying search system. In this paper we carry out a user study to systematically investigate how click behaviour changes for different levels of search system effectiveness as measured by information retrieval performance metrics. Our results show that click behaviour does not vary systematically with the quality of search results. However, click behaviour does vary significantly between individual users, and between search topics. This suggests that using direct click behaviour--click rank and click frequency--to infer the quality of the underlying search system is problematic. Further analysis of our user click data indicates that the correspondence between clicks in a search result list and subsequent confirmation that the clicked resource is actually relevant is low. Using clicks as an implicit indication of relevance should therefore be done with caution.

References

[1]

Agichtein, E., Brill, E., Dumais, S.: Improving web search ranking by incorporating user behavior information. In: Efthimiadis, et al. (eds.) {7}, pp. 19-26.

Digital Library

[2]

Agichtein, E., Brill, E., Dumais, S., Ragno, R.: Learning user interaction models for predicting web search result preferences. In: Efthimiadis, et al. (eds.) {7}, pp. 3-10.

Digital Library

[3]

Allan, J., Carterette, B., Lewis, J.: When will information retrieval be "good enough"? In: Marchionini, et al. (eds.) {15}, pp. 433-440.

Digital Library

[4]

Bailey, P., Craswell, N., Hawking, D.: Engineering a multi-purpose test collection for web retrieval experiments. Information Processing and Management 39(6), 853-871 (2003).

Digital Library

[5]

Buckley, C., Voorhees, E.M.: Retrieval system evaluation. In: TREC: experiment and evaluation in information retrieval {21}.

[6]

Craswell, N., Szummer, M.: Random walks on the click graph. In: Kraaij, et al. (eds.) {14}, pp. 239-246.

Digital Library

[7]

Efthimiadis, E., Dumais, S., Hawking, D., Järvelin, K. (eds.): Proceedings of the 29th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Seattle, WA (2006).

[8]

Fox, S., Karnawat, K., Mydland, M., Dumais, S., White, T.: Evaluating implicit measures to improve web search. ACM Transactions on Information Systems 23(2), 147-168 (2005).

Digital Library

[9]

Harman, D.K.: The TREC test collection. In: TREC: experiment and evaluation in information retrieval {21}.

[10]

Joachims, T.: Optimizing search engines using clickthrough data. In: Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, Edmonton, Alberta, Canada, pp. 133-142. ACM Press, New York (2002).

Digital Library

[11]

Joachims, T., Granka, L., Pan, B., Hembrooke, H., Gay, G.: Accurately interpreting clickthrough data as implicit feedback. In: Marchionini, et al. (eds.) {15}, pp. 154-161.

Digital Library

[12]

Joachims, T., Granka, L., Pan, B., Hembrooke, H., Radlinski, F., Gay, G.: Evaluating the accuracy of implicit feedback from clicks and query reformulations in web search. ACM Transactions on Information Systems 25(2), 7 (2007).

Digital Library

[13]

Kemp, C., Ramamohanarao, K.: Long-term learning for web search engines. In: Proceedings of the 6th European Conference on Principles of Data Mining and Knowledge Discovery, London, UK, pp. 263-274. Springer, Heidelberg (2002).

Digital Library

[14]

Kraaij, W., de Vries, A., Clarke, C., Fuhr, N., Kando, N. (eds.): Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Amsterdam, The Netherlands (2007).

[15]

Marchionini, G., Moffat, A., Tait, J., Baeza-Yates, R., Ziviani, N. (eds.): Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Salvador, Brazil (2005).

[16]

Radlinski, F., Joachims, T.: Query chains: learning to rank from implicit feedback. In: Proceeding of the eleventh ACM SIGKDD international conference on Knowledge discovery in data mining, Chicago, Illinois, USA, pp. 239-248 (2005).

Digital Library

[17]

Radlinski, F., Joachims, T.: Active exploration for learning rankings from click-through data. In: Proceedings of the 13th ACM SIGKDD international conference on Knowledge discovery and data mining, San Jose, California, pp. 570-579 (2007).

Digital Library

[18]

Turpin, A., Scholer, F.: User performance versus precision measures for simple search tasks. In: Efthimiadis, et al. (eds.) {7}, pp. 11-18.

Digital Library

[19]

Turpin, A., Scholer, F., Billerbeck, B., Abel, L.: Examining the pseudostandard web search engine results page. In: Proceedings of the 11th Australasian Document Computing Symposium, Brisbane, Australia, pp. 9-16 (2006).

[20]

Turpin, A., Tsegay, Y., Hawking, D., Williams, H.E.: Fast generation of result snippets in web search. In: Kraaij, et al. (eds.) {14}, pp. 127-134.

Digital Library

[21]

Voorhees, E.M., Harman, D.K.: TREC: experiment and evaluation in information retrieval. MIT Press, Cambridge (2005).

Digital Library

Cited By

Zoghi MWhiteson Sde Rijke MCheng XLi HGabrilovich ETang J(2015)MergeRUCBProceedings of the Eighth ACM International Conference on Web Search and Data Mining10.1145/2684822.2685290(17-26)Online publication date: 2-Feb-2015
https://dl.acm.org/doi/10.1145/2684822.2685290
Zoghi MWhiteson Sde Rijke MMunos RCarterette BDiaz FCastillo CMetzler D(2014)Relative confidence sampling for efficient on-line ranker evaluationProceedings of the 7th ACM international conference on Web search and data mining10.1145/2556195.2556256(73-82)Online publication date: 24-Feb-2014
https://dl.acm.org/doi/10.1145/2556195.2556256
Hofmann KWhiteson SRijke M(2013)Fidelity, Soundness, and Efficiency of Interleaved Comparison MethodsACM Transactions on Information Systems10.1145/2536736.253673731:4(1-43)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1145/2536736.2536737
Show More Cited By

Using clicks as implicit judgments: expectations versus observations
1. Information systems
  1. Information retrieval

Recommendations

Evaluating implicit judgments from image search clickthrough data

The interactions of users with search engines can be seen as implicit relevance feedback by the user on the results offered to them. In particular, the selection of results by users can be interpreted as a confirmation of the relevance of those results, ...
Predicting clicks: estimating the click-through rate for new ads
WWW '07: Proceedings of the 16th international conference on World Wide Web

Search engine advertising has become a significant element of the Web browsing experience. Choosing the right ads for the query and the order in which they are displayed greatly affects the probability that a user will see and click on each ad. This ...
Exploring relevance for clicks
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Mining feedback information from user click-through data is an important issue for modern Web retrieval systems in terms of architecture analysis, performance evaluation and algorithm optimization. For commercial search engines, user click-through data ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ECIR'08: Proceedings of the IR research, 30th European conference on Advances in information retrieval

March 2008

718 pages

ISBN:3540786457

Editors:
Craig Macdonald
University of Glasgow, Department of Computing Science, Glasgow, UK
,
Iadh Ounis
University of Glasgow, Department of Computing Science, Glasgow, UK
,
Vassilis Plachouras
Yahoo! Research, Barcelona, Spain
,
Ian Ruthven
University of Strathclyde, Department of Computing and Information Sciences, Glasgow, UK
,
Ryen W. White
Microsoft Research, Redmond, WA

Sponsors

Yahoo! Research
Google Inc.
Microsoft Research: Microsoft Research
Matrixware Information Services

In-Cooperation

SIGIR: ACM Special Interest Group on Information Retrieval
British Computer Society: BCS

Publisher

Springer-Verlag

Berlin, Heidelberg

Publication History

Published: 30 March 2008

Check for updates

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

10
Total Citations
View Citations
12
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zoghi MWhiteson Sde Rijke MCheng XLi HGabrilovich ETang J(2015)MergeRUCBProceedings of the Eighth ACM International Conference on Web Search and Data Mining10.1145/2684822.2685290(17-26)Online publication date: 2-Feb-2015
https://dl.acm.org/doi/10.1145/2684822.2685290
Zoghi MWhiteson Sde Rijke MMunos RCarterette BDiaz FCastillo CMetzler D(2014)Relative confidence sampling for efficient on-line ranker evaluationProceedings of the 7th ACM international conference on Web search and data mining10.1145/2556195.2556256(73-82)Online publication date: 24-Feb-2014
https://dl.acm.org/doi/10.1145/2556195.2556256
Hofmann KWhiteson SRijke M(2013)Fidelity, Soundness, and Efficiency of Interleaved Comparison MethodsACM Transactions on Information Systems10.1145/2536736.253673731:4(1-43)Online publication date: 1-Nov-2013
https://dl.acm.org/doi/10.1145/2536736.2536737
Antunovic MCaon GTruran MAshman HStumme GHotho A(2013)Discovering semantic associations from web search interactionsProceedings of the 24th ACM Conference on Hypertext and Social Media10.1145/2481492.2481517(202-207)Online publication date: 1-May-2013
https://dl.acm.org/doi/10.1145/2481492.2481517
Hofmann KBehr FRadlinski FChen XLebanon GWang HZaki M(2012)On caption bias in interleaving experimentsProceedings of the 21st ACM international conference on Information and knowledge management10.1145/2396761.2396780(115-124)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2396761.2396780
Hofmann KWhiteson Sde Rijke M(2011)A probabilistic method for inferring preferences from clicksProceedings of the 20th ACM international conference on Information and knowledge management10.1145/2063576.2063618(249-258)Online publication date: 24-Oct-2011
https://dl.acm.org/doi/10.1145/2063576.2063618
Ashman HAntunovic MChaprasit SSmith GTruran MDe Bra PGrønbæk K(2011)Implicit association via crowd-sourced coselectionProceedings of the 22nd ACM conference on Hypertext and hypermedia10.1145/1995966.1995972(7-16)Online publication date: 6-Jun-2011
https://dl.acm.org/doi/10.1145/1995966.1995972
Jain VVarma MSadagopan SRamamritham KKumar ARavindra MBertino EKumar R(2011)Learning to re-rankProceedings of the 20th international conference on World wide web10.1145/1963405.1963447(277-286)Online publication date: 28-Mar-2011
https://dl.acm.org/doi/10.1145/1963405.1963447
Ashman HAntunovic MDonner CFrith RRebelos ESchmakeit JSmith GTruran M(2009)Are Clickthroughs Useful for Image Labelling?Proceedings of the 2009 IEEE/WIC/ACM International Joint Conference on Web Intelligence and Intelligent Agent Technology - Volume 0110.1109/WI-IAT.2009.35(191-197)Online publication date: 15-Sep-2009
https://dl.acm.org/doi/10.1109/WI-IAT.2009.35
Luxenburger JElbassuoni SWeikum GShanahan JAmer-Yahia SManolescu IZhang YEvans DKolcz AChoi KChowdury A(2008)Matching task profiles and user needs in personalized web searchProceedings of the 17th ACM conference on Information and knowledge management10.1145/1458082.1458175(689-698)Online publication date: 26-Oct-2008
https://dl.acm.org/doi/10.1145/1458082.1458175

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

Media

Figures

Other

Tables

View Table of Contents