article

An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine

Authors:

Panos Balatsoukas,

Ian RuthvenAuthors Info & Claims

Journal of the American Society for Information Science and Technology, Volume 63, Issue 9

Pages 1728 - 1746

https://doi.org/10.1002/asi.22707

Published: 01 September 2012 Publication History

Abstract

Eye movement data can provide an in-depth view of human reasoning and the decision-making process, and modern information retrieval (IR) research can benefit from the analysis of this type of data. The aim of this research was to examine the relationship between relevance criteria use and visual behavior in the context of predictive relevance judgments. To address this objective, a multimethod research design was employed that involved observation of participants’ eye movements, talk-aloud protocols, and postsearch interviews. Specifically, the results reported in this article came from the analysis of 281 predictive relevance judgments made by 24 participants using the Google search engine. We present a novel stepwise methodological framework for the analysis of relevance judgments and eye movements on the Web and show new patterns of relevance criteria use during predictive relevance judgment. For example, the findings showed an effect of ranking order and surrogate components (Title, Summary, and URL) on the use of relevance criteria. Also, differences were observed in the cognitive effort spent between very relevant and not relevant judgments. We conclude with the implications of this study for IR research. © 2012 Wiley Periodicals, Inc.

References

[1]

Aula, A., Majaranta, P., & Raiha, K-J. (2005). Eye-tracking reveals the personal styles for search result evaluation. In M.F. Constabile & F. Paterno (Eds.), INTERACT 2005 (pp. 1058––1061). Berlin: Springer-Verlag.

[2]

Balatsoukas, P., Morris, A., & O'Brien, A. (2009). An evaluation framework of user interaction with metadata surrogates. Journal of Information Science, 35(3), 321–339.

Digital Library

[3]

Balatsoukas, P., & Ruthven I. (2010). The use of relevance criteria during predictive judgment: An eye tracking approach. Proceedings of the American Society for Information Science and Technology, 47(1), 1–10.

[4]

Barry, C., & Schamber, L. (1998). Users’’ criteria for relevance evaluation: A cross-situational comparison. Information Processing and Management, 34(2/3), 219–236.

Digital Library

[5]

Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society of Information Science and Technology, 54(10), 913–925.

Digital Library

[6]

Borlund, P., & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3), 225–250.

[7]

Buscher, G., Cutrell, E., & Morris, M. (2009). What do you see when you're surfing? Using eye tracking to predict salient regions of Webpages. In Proceedings of the 27^th International Conference on Human Factors in Computing Systems (pp. 21–30). New York: ACM.

[8]

Crystal, A., & Greenberg, J. (2006). Relevance criteria identified by health information users during Web searches. Journal of the American Society for Information Science and Technology, 57(10), 1368–2882.

Digital Library

[9]

Cutrell, E., & Guan, Z. (2007). What are you looking for? An eye-tracking study of information usage in Web search. In Proceedings of the ACM HCI 2007 Conference on Human Factors in Computing Systems (pp. 407–415). New York: ACM.

[10]

Duchowski, A. (2007). Eye tracking methodology: Theory and practice. Berlin: Springer-Verlag.

[11]

Dumais, S., Buscher, G., & Cutrell, E. (2010). Individual differences in gaze patterns for Web search. In Proceedings of the Third Symposium on Information Interaction in Context (pp. 185–194). New York: ACM.

[12]

Eisenberg, M.B. (1988). Measuring relevance judgments. Information Processing & Management, 24(4), 373––389.

Digital Library

[13]

Ericsson, A., & Simon, H. (1993). Protocol analysis: Verbal reports as data. Cambridge, MA: MIT Press.

[14]

Foster, A., & Ford, N. (2003). Serendipity and information seeking: An empirical study. Journal of Documentation, 59, 321––340.

[15]

Gerjets, P., Kammerer, Y., & Werner, B., (2011). Measuring spontaneous and instructed evaluation processes during Web search: Integrating concurrent thinking aloud protocols and eye tracking data. Learning and Instruction, 21(2), 220–231.

[16]

Goldberg, J.H., Stimson, M., Lewenstein, M., Scott, N., & Wichansky, A. (2002). Eye Tracking in Web search tasks: Design implications. In Proceedings of the Eye tracking Research and Applications Symposium (ETRA) (pp. 51–58). New York: ACM.

[17]

Granka, L., Joachims, T., & Gay, G. (2004). Eye-tracking analysis of user behavior in WWW search. In M. Sanderson, K. Jarvelin, J. Allan, & P. Bruza (Eds.), Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (pp. 478–479). New York: ACM.

[18]

Hughes, A., Wilkens, T., Wildemuth, B., & Marchionini, G. (2003). Text or pictures? An eye tracking study of how people view digital video surrogates. In E.M. Bakker, T.S. Huang, M.S. Lew, N. Sebe, & X.S. Zhou (Eds.), Proceedings of the Second International Conference CIVR 2003 Urbana-Champaign, IL, July 24––25 (pp. 271–280). Berlin: Springer-Verlag.

[19]

Lorigo, L, Haridasan, M., Brynjarsdottir, H., Xia, L., Joachims, T., Gay, G., … Pan, B. (2008). Eye tracking and online search: Lessons learned and challenges ahead. Journal of the American Society for Information Science and Technology, 59(7), 1041–1052.

Digital Library

[20]

Lorigo, L., Pan, B., Hembrooke, H., Joachims, T., Granka, L., & Gay, G. (2006). The influence of task and gender on search and evaluation behavior using Google. Information Processing and Management, 42, 1123–1131.

Digital Library

[21]

Pan, B., Hembrooke, H., Joachims, T., Lorigo, L., Gay, G., & Granka, L. (2007). In Google we trust: Users’’ decisions on rank, position and relevance. Journal of Computer-Mediated Communication, 12, 801–823.

[22]

Papaeconomou, C., Zijlema, A., & Ingwersen, P. (2008). Searchers’’ relevance judgments and criteria in evaluating Web pages in a learning style perspective. In Proceedings of the Second international symposium on Information Interaction in Context (IIX 2008) (pp. 123–132). New York: ACM Press.

[23]

Rayner, K. (2009). Eye movements and attention in reading, scene perception and visual search. The Quarterly Journal of Experimental Psychology, 62, 1457–1506.

[24]

Rele, S., & Duchowski, A.T. (2005). Using eye tracking to evaluate alternative search results interfaces. Human Factors and Ergonomics Society Annual Meeting, 49(15), 1459–1463.

[25]

Rieh, S.Y. (2002). Judgment of information quality and cognitive authority in the web. Journal of the American Society for Information Science and Technology, 53(2), 145–161.

[26]

Rodden, K., Fu, X., Aula, A., & Spiro, I. (2008). Eye-mouse coordination patterns on Web search pages. In Proceedings of the ACM CHI 2005 Conference on Human Factors in Computing Systems (pp. 2997–3002). New York: ACM Press.

[27]

Ruthven, I. (2008). Interactive information retrieval. Annual Review of Information Science and Technology, 42, 43–92.

[28]

Saracevic, T., & Su, L. (1989). Modeling and measuring user-intermediary computer interaction in online searching: Design of a study. Proceedings of the Annual Meeting of the American Society for Information Science and Technology, 26, 75–80.

[29]

Savolainen, R., & Kari, J. (2006). User-defined relevance criteria in Web searching. Journal of Documentation, 62(6), 685––707.

[30]

Schamber, L., Eisenberg, M., & Nilan, M. (1990). A re-examination of relevance: Towards a dynamic, situational definition. Journal of Information Processing and Management, 26(6), 755–776.

Digital Library

[31]

Spink, A., & Greisdorf, H. (2001). Regions and levels: Measuring and mapping users’’ relevance judgments. Journal of the American Society for Information Science and Technology, 52(2), 161–173.

Digital Library

[32]

Spink, A., Greisdorf, H., & Bateman, J. (1998). From highly relevant to not relevant: Examining different regions of relevance. Information Processing and Management, 34(5), 599–621.

Digital Library

[33]

Tang, R., & Solomon, P. (2001) Use of relevance criteria across stages of document evaluation: On the complementarity of experimental and naturalistic studies, Journal of the American Society for Information Science and Technology, 52(8), 676–685.

Digital Library

[34]

Tombros, A., Ruthven, I., & Jose, J. (2005). How users’’ assess Web pages for information seeking. Journal of the American society for Information Science and Technology, 56(4), 327–344.

Digital Library

[35]

Wang, P., & Soergel, D. (1998). A cognitive model of document use during a research project. Study I. Document selection. Journal of the American Society for Information Science and Technology, 49(2), 115–133.

Digital Library

[36]

Zhang, Y., & Yuelin, L. (2008). A user-centered functional metadata evaluation of moving image collections. Journal of the American Society for Information Science and Technology, 59(8), 1331–1346.

Digital Library

Cited By

Salminen JJung SJansen B(2024)Investigating Persona Viewing Behavior: An Eye-Tracking Study on Portrait-Format Persona ProfileProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685376(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685376
Choi BCasteel SArguello JCapra R(2023)Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and ChallengesACM Transactions on Information Systems10.1145/363000442:3(1-32)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3630004
Dong SGoldstein JYang GCrestani FPasi GGaussier E(2022)GazBy: Gaze-Based BERT Model to Incorporate Human Attention in Neural Information RetrievalProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545129(182-192)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3539813.3545129
Show More Cited By

An eye-tracking approach to the analysis of relevance judgments on the Web: The case of Google search engine
1. Information systems

Recommendations

The use of relevance criteria during predictive judgment: an eye tracking approach
ASIS&T '10: Proceedings of the 73rd ASIS&T Annual Meeting on Navigating Streams in an Information Ecosystem - Volume 47

This paper reports on the findings of a user study that explored how searchers tend to fixate on information associated with different relevance criteria in the web during the relevance judgment process. The user study involved the completion of ...
Understanding Relevance Judgments in Legal Case Retrieval
Legal case retrieval, which aims to retrieve relevant cases given a query case, has drawn increasing research attention in recent years. While much research has worked on developing automatic retrieval models, how to characterize relevance in this ...
Graphical representation and similarity measurement of relevance judgments on the web
IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

The purpose of this paper is to present a method for the graphical representation and similarity measurement of relevance judgments on the web. In order to address this objective a Latent Semantic Indexing technique was used. The findings suggest that ...

Comments

Information & Contributors

Information

Published In

cover image Journal of the American Society for Information Science and Technology

Journal of the American Society for Information Science and Technology Volume 63, Issue 9

September 2012

210 pages

ISSN:1532-2882

Issue’s Table of Contents

Publisher

John Wiley & Sons, Inc.

United States

Publication History

Published: 01 September 2012

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

23
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Salminen JJung SJansen B(2024)Investigating Persona Viewing Behavior: An Eye-Tracking Study on Portrait-Format Persona ProfileProceedings of the 13th Nordic Conference on Human-Computer Interaction10.1145/3679318.3685376(1-12)Online publication date: 13-Oct-2024
https://dl.acm.org/doi/10.1145/3679318.3685376
Choi BCasteel SArguello JCapra R(2023)Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and ChallengesACM Transactions on Information Systems10.1145/363000442:3(1-32)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3630004
Dong SGoldstein JYang GCrestani FPasi GGaussier E(2022)GazBy: Gaze-Based BERT Model to Incorporate Human Attention in Neural Information RetrievalProceedings of the 2022 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3539813.3545129(182-192)Online publication date: 23-Aug-2022
https://dl.acm.org/doi/10.1145/3539813.3545129
Abualsaud MSmucker MClarke CScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Visualizing Searcher Gaze PatternsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446041(295-299)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446041
Feit AVordemann LPark SBerube CHilliges O(2020)Detecting Relevance during Decision-Making from Eye Movements for UI AdaptationACM Symposium on Eye Tracking Research and Applications10.1145/3379155.3391321(1-11)Online publication date: 2-Jun-2020
https://dl.acm.org/doi/10.1145/3379155.3391321
Behnert CAzzopardi LHalvey MRuthven IJoho HMurdock VQvarfordt P(2019)Investigating the Effects of Popularity Data on Predictive Relevance Judgments in Academic Search SystemsProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298978(437-440)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3295750.3298978
Hienert DKern DMitsui MShah CBelkin NAzzopardi LHalvey MRuthven IJoho HMurdock VQvarfordt P(2019)Reading ProtocolProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298921(73-81)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3295750.3298921
Connaway LFaniel INarayan BSayyed‐Abdi E(2019)There is a method to itProceedings of the Association for Information Science and Technology10.1002/pra2.8056:1(521-524)Online publication date: 18-Oct-2019
https://dl.acm.org/doi/10.1002/pra2.80
Zhang GWang JZhou GLiu JWei CEmrouznejad AQian Z(2018)Scientific Data Relevance Criteria Classification and UsageProceedings of the 2nd International Conference on Computer Science and Application Engineering10.1145/3207677.3278010(1-7)Online publication date: 22-Oct-2018
https://dl.acm.org/doi/10.1145/3207677.3278010
Salminen JJansen BAn JJung SNielsen LKwak HShah CBelkin NByström KHuang JScholer F(2018)Fixation and ConfusionProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176391(110-119)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1145/3176349.3176391
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents