research-article

The Usefulness of Search Results: A Systematization of Types and Predictors

Author:

Pertti VakkariAuthors Info & Claims

CHIIR '20: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval

Pages 243 - 252

https://doi.org/10.1145/3343413.3377955

Published: 14 March 2020 Publication History

Abstract

In evaluating search systems there is a growing trend to complement the established effectiveness indicator topical relevance by the usefulness of search results. Usefulness refers to the contribution of search results to a larger task generating information search. This study analyses articles on interactive information retrieval which either predict the usefulness of documents retrieved or evaluate search systems by the usefulness of search results. The aim is to systematize the findings of these studies by categorizing the types of usefulness and their predictor types. Significant empirical associations between predictor types and usefulness are systematized, too. The data consists of articles in journals or conference proceedings focusing on the usefulness of search results either in web or database environment. The results indicate that there is a growing trend to complement topical relevance by the usefulness of search results in evaluating search systems. Search tasks typically instruct participants to search information for a writing task. Perceived usefulness of search results is the established measure in studies, although there are alternative measures in use. Significant associations between predictors and usefulness do not accumulate much but vary notably. Growth of knowledge on the usefulness of search results is based on the increasing number and variety of proposition supported empirically.

References

[1]

CORPUS

[2]

M. Ageev, Q. Guo, D. Lagun, and E. Agichtein (2011). Find it if you can: A game for modeling different types of web search success using interaction data. In Proc. of SIGIR'11, ACM, New York, NY, 345--354.

Digital Library

[3]

S. Akuma, R. Iqbal, C. Jayne, and F. Doctor (2016). Comparative analysis of relevance feedback method based on two user studies. Computers in Human Behavior 60, 138--146.

Digital Library

[4]

J-W. Ahn, P. Brusilovsky, D. He, J. Grady, and Q. Li. (2008). Personalized web exploration with task models. In Proc. WWW 2008. ACM, New York, NY, 1--10.

Digital Library

[5]

N. Bhattacharya, and J. Gwizdka (2019). Measuring learning during search: Differences in interactions, eye-gaze, and semantic similarity to expert knowledge. In Proc CHIIR'19, New York, NY: ACM, 63--71.

Digital Library

[6]

M. Bron, J. van Gorp, F. Nack, M. de Rijke, A. Vishneuski, and S. de Leeuw S. (2012). A subjunctive exploratory search interface to support media studies researchers. In Proc. SIGIR'12. New York, NY, ACM, 425--434.

Digital Library

[7]

J. Budzik, and K. Hammond (2000). User interactions with everyday applications as context for just-in-time information access. In Proc. UIU'00. New York, NY, ACM, 44--51.

Digital Library

[8]

K.R. Butcher, S. Davies, A. Crockett, A. Dewald, and R. Zheng (2011). Do graphical search interfaces support effective search for and evaluation of digital library resources? In Proc. JCDL'11. New York, NY, ACM, 2011, 315--324.

Digital Library

[9]

K. Collins-Thompson, S.Y. Rieh, C.C. Haynes, and R. Syed (2016). Assessing learning outcomes in web search: A comparison of tasks and query strategies. Proc. CHIIR'16. New York, NY, ACM, 163--172.

Digital Library

[10]

S. Ghosh, M. Rath, and C. Shah (2018). Searching as learning: Exploring search behavior and learning outcomes in learning-related tasks. In Proc. CHIIR'18. ACM, New York, 22--31.

Digital Library

[11]

Q. Guo, and E. Agichtein (2012). Beyond dwell time: Estimating document relevance from cursor movements and other post-click search behavior. In Proc. of the 21th WWW conference. ACM, New York, NY, 569--578.

Digital Library

[12]

Q. Guo, D. Lagun, and E. Agichtein (2012). Predicting Web search success with fine-grained interaction data. In Proc. CIKM'12. ACM, New York, NY, 2050--2054.

Digital Library

[13]

D. He, P. Brusilovsky, J. Ahn, J. Grady, R. Farzan, Y. Peng, Y. Yang, and M. Rogati (2008). An evaluation of adaptive filtering in the context of realistic task-based information exploration. IPM 44, 511--533.

[14]

W. Hersh, J. Pentecost, and D. Hickam (1996). A task-oriented approch to information retrieval. JASIST 47(1), 50--56.

[15]

J. Jiang, D. He, and J. Allan (2017a). Comparing in situ and multidimensional relevance judgments. Proc. SIGIR'17. New York, NY, ACM, 405--414.

Digital Library

[16]

J. Jiang, D. He, D. Kelly, and J. Allan (2017b) Understanding ephemeral state of relevance. In Proc. CHIIR'17, New York, NY, ACM, 137--146.

Digital Library

[17]

D. Kelly, and N. J. Belkin (2004). Display time as implicit feedback: Understanding task effects. In Proc. SIGIR'04. New York, NY, ACM, 377--384.

Digital Library

[18]

J. Kim, J. Teevan, and N. Crasswell (2016). Explicit in situ user feedback for Web search results. In Proc. SIGIR'16. New York, NY, ACM, 829--832.

Digital Library

[19]

C. Liu, N.J. Belkin, and M.J. Cole (2012). Personalization of search results using interaction behaviors in search sessions. In Proc. SIGIR'12. New York, NY, ACM, 205--214.

Digital Library

[20]

J. Liu, and N. J. Belkin (2010). Personalizing information retrieval for multi-session tasks: the roles of task stage and task type. In Proc. SIGIR'10. New York, NY, ACM, 26--33.

Digital Library

[21]

J. Liu, and N. J. Belkin. (2012). Searching vs. writing: Factors affecting information use task performance. In Proc. ASIST, Vol. 49, 1, 1--10.

[22]

M. Liu, Y. Liu, X. Mao, C. Luo, M. Zhang, and S. Ma (2018), "Satisfaction with failure" or "unsatisfied success": Investigating the relationship between search success and user satisfaction. In Proc. WWW 2018. New York, NY, ACM, 1533--1542.

Digital Library

[23]

M. Lykke, S. Price, L. Delacambre, and P. Vedsted (2012). How doctors search: A study of family practitioners' query behavior and the impact on search results. IPM 48(6), 1151--1170.

[24]

J. Mao, Y. Liu, H. Luan, M. Zhang, S. Ma, and H. Luo (2017). Understanding and predicting usefulness judgment in web search. Proc. SIGIR'17. New York, NY, ACM, 1169--1172.

Digital Library

[25]

J. Mao, Y. Liu, K. Zhou, J. Nie, M. Zhang, and S. Ma (2016). When does relevance mean usefulness and user satisfaction in web search. In Proc. SIGIR'16. New York, NY, ACM, 463--472.

Digital Library

[26]

M. Pennanen, and P. Vakkari (2003). Students' conceptual structure, search process and outcome while preparing a research proposal. JASIST 54(8), 759--770.

Digital Library

[27]

T. Ruotsalo, J. Peltonen, M. Eugster, D. Glowacka, K. Konyushkova, K. Athukorala, I. Kosunen, A. Reijonen, P. Myllymäki, G. Jacucci, and S. Kaski (2013). Directing exploratory search with interactive intent modeling. In Proc. CIKM'13. ACM, New York, NY, 1759--1764.

Digital Library

[28]

S. Rutter, V. Blinzler, C. Ye, and M.L. Wilson (2019), Search tactics used in solving everyday how-to technical tasks: Repertoire, selection and tenacity. IPM 56, 919--938.

[29]

X. Song, H. Liu, and C. Liu, (2018). Characterizing and exploring users' task completion process at different stages in learning related tasks. In Proc. ASIST Annual Meeting 2018, 460--469.

[30]

R. Syed, and K. Collins-Thompson (2017). Retrieval algorithms optimized for human learning. In Proc. SIGIR'17. New York, NY, ACM, 555--564.

Digital Library

[31]

P. Vakkari (2000). Relevance and contributing information types of searched documents in task performance. In Proc. SIGIR'00. New York, NY, ACM, 2--9.

Digital Library

[32]

P. Vakkari, and S. Huuskonen (2012). Search effort degrades search output but improves task outcome. JASIST 63(4), 657--670.

[33]

P. Vakkari, M. Völske, M. Potthast, M. Hagen, and B. Stein (2018). Predicting Retrieval Success Based on Information Use for Writing Tasks. In Proc. TPDL 2018, Lecture Notes in Computer Science, vol. 11057. Springer, Cham, 161--173.

[34]

P. Vakkari, M. Völske, M. Potthast, M. Hagen, and B. Stein (2019). Modeling the usefulness of search results as measured by information use. IPM 56(3), 879--894.

[35]

B.M. Wildemuth, R. de Bliek, C.P. Friedman, and D.D. File (1995). Medical students' personal knowledge, searching proficiency, and database use in problem solving. JASIST 46(8), 590--607.

Digital Library

[36]

I-C. Wu (2011). Toward supporting information-seeking and retrieval activities based on evolving topic needs. J.Doc 67(3), 525--561 REFERENCES

[37]

O. Alonso, and S. Mizzaro (2012). Using crowdsourcing for TREC relevance assessment. IPM 48(6), 1053--1066.

[38]

N.J. Belkin, M. Cole, and J. Liu (2009). A model for evaluating interactive information retrieval. In SIGIR Workshop on the Future of IR Evaluation, July 23, 2009, Boston.

[39]

P. Borlund (2000). Evaluation of interactive information retrieval systems. Diss. Åbo Akademi University Press, Åbo.

[40]

W. S. Cooper (1973). On selecting a measure of retrieval effectiveness. JASIST 24( 2), 87--100.

[41]

C. Cuadra, and R. Katter (1967). Opening the black box of relevance. J.Doc. 23, 291--303.

[42]

S. Harter (1992). Psychological relevance and information science. JASIST 43(9), 602--615.

[43]

S.W. Hersh (2003). Information retrieval: A health and biomedical perspective (2nd ed.). New York, Springer.

[44]

J. Jiang, A. Hassan, X. Shi, and R. White (2015). Understanding and predicting graded search satisfaction. In Proc. SIGIR'15. ACM, New York, 57--66.

Digital Library

[45]

K. Ja¨rvelin, P. Vakkari, P. Arvola, F. Baskaya, A. J¨arvelin, J. Kek¨ala¨inen, H. Keskustalo, S. Kumpulainen, M. Saastamoinen, R. Savolainen, and E. Sormunen (2015) Task-based information interaction evaluation: The viewpoint of program theory. ACM Trans. Inf. Syst. 33(1):3:1--3:30.

Digital Library

[46]

D. Kelly, J. Arguello, A. Edwards, and W-C. Wu (2015). Development and evaluation of search tasks for IIR experiments using a cognitive complexity framework. In Proc. ICTIR'15. New York, NY, ACM, 101--110.

Digital Library

[47]

C. Kuhlthau (1993). Seeking Meaning. Norwood, N.J.: Ablex.

[48]

W. Lancaster, and A. Warner A. (1993). Information retrieval today. Information Resources Press: Arlington, VA.

Digital Library

[49]

J. Liu, M. Cole, R. Bierig, J. Gwizdka, and N.J. Belkin (2010). Search behavior in different task types. In Proc. JCDL'10. ACM, New York, 69--78.

Digital Library

[50]

A. Moffat, P. Thomas, and F. Scholer, (2013). Users versus models: What observation tells us about effectiveness metrics. In Proc. CIKM´13. ACM, New York, 659--668.

Digital Library

[51]

A. Rees (1967). Evaluation of information systems and services. ARIST 2, 63--86.

[52]

S.Y. Rieh, J. Gwizdka, L. Freund, and K. Collins-Thompson (2014). Searching as Learning: Novel measures for information interaction research. In Proc. 77th ASIST Annual Meeting, 1--4.

[53]

W. Rouse, and S. Rouse (1983). Human information seeking and the design of information systems. IPM 20(1--2), 129--138.

[54]

T. Saracevic (1975). Relevance: A review of and a framework for thinking on the notion in information science. JASIST 26(6), 321--343.

[55]

L. Schamber, M. Eisenberg, and M. Nilan (1990). A re-examination of relevance: towards a dynamic, situational definition. IPM 26, 755--776.

[56]

C. Shah, and R. Gonzalez-Ibanes (2011). Evaluating the synergic effect of collaboration on information seeking. In Proc. SIGIR'11, ACM, New York, 913--922.

Digital Library

[57]

D. Silverman (2002). Doing qualitative research. Sage: London.

[58]

A. Stinchcombe (1968). Constructing social theories. New York, Harcourt.

[59]

E. Toms, R. Villa, and L. McCay-Peet (2013). How is search system used in work task completion. J. Inf. Sci. 39(1), 15--25.

Digital Library

[60]

P. Vakkari (2001). A theory of the task-based information retrieval process: A summary and generalization of a longitudinal study. J.Doc 57(1), 44--60.

[61]

P. Vakkari (1998). Growth of theories on information seeking. IPM 34(2--3), 361--382.

[62]

P. Vakkari (2016). Searching as learning. A Systematization based on literature. J. Inf. Sci. 42(1), 7--18.

Digital Library

[63]

P. Vakkari (2003) Task-based information searching. ARIST 2003, vol. 37. Information Today: Medford, NJ, 413--464.

[64]

D. Wagner, and J. Berger (1985). Do sociological theories grow? Am. J.Soc. 90, 697--728.

[65]

J. Walhout, P. Oomen, J. Halszka, and S. Brand-Gruwel (2017). Effects of task complexity on online search behavior of adolescents. JASIST 68(6), 1449--1461.

[66]

B. Wildemuth, C. Friedman, J. Keyes, and S. Downs (2000), A longitudinal study of database-assisted problem solving. IPM 36(3), 445--459

[67]

R. Yu, U. Gadiraju, P. Holtz, M. Rokicki, P. Kemkes, and S. Dietze (2018). Predicting user knowledge gain in informational search sessions. In Proc. SIGIR'18. New York, NY, ACM, 75--84.

Digital Library

Cited By

Schoegje THardman LDe Vries APieters T(2024)Improving expert search effectiveness: Comparing ways to rank and present search resultsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638296(56-65)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638296
Peikos GPasi G(2024)A systematic review of multidimensional relevance estimation in information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.154114:5Online publication date: 7-May-2024
https://doi.org/10.1002/widm.1541
Van der Sluis FFaure JHomnual S(2024)An empirical exploration of the subjectivity problem of information qualitiesJournal of the Association for Information Science and Technology10.1002/asi.24884Online publication date: 25-Mar-2024
https://doi.org/10.1002/asi.24884
Show More Cited By

Index Terms

The Usefulness of Search Results: A Systematization of Types and Predictors
1. Information systems
  1. Information retrieval
    1. Evaluation of retrieval results
      1. Retrieval effectiveness
    2. Users and interactive retrieval

Recommendations

Viewpoint Diversity in Search Results
Advances in Information Retrieval
Abstract
Adverse phenomena such as the search engine manipulation effect (SEME), where web search users change their attitude on a topic following whatever most highly-ranked search results promote, represent crucial challenges for research and industry. ...
Personalization of search results using interaction behaviors in search sessions
SIGIR '12: Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval

Personalization of search results offers the potential for significant improvement in information retrieval performance. User interactions with the system and documents during information-seeking sessions provide a wealth of information about user ...
A graph-theoretic approach to extract storylines from search results
KDD '04: Proceedings of the tenth ACM SIGKDD international conference on Knowledge discovery and data mining

We present a graph-theoretic approach to discover storylines from search results. Storylines are windows that offer glimpses into interesting themes latent among the top search results for a query; they are different from, and complementary to, clusters ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

CHIIR '20: Proceedings of the 2020 Conference on Human Information Interaction and Retrieval

March 2020

596 pages

ISBN:9781450368926

DOI:10.1145/3343413

General Chairs:
Heather O'Brien
University of British Columbia, Canada
,
Luanne Freund
University of British Columbia, Canada
,
Program Chairs:
Ioannis Arapakis
Telefonica I+D, Spain
,
Orland Hoeber
University of Regina, Canada
,
Irene Lopatovska
Pratt Institute, USA

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 14 March 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CHIIR '20

Sponsor:

CHIIR '20: Conference on Human Information Interaction and Retrieval

March 14 - 18, 2020

Vancouver BC, Canada

Acceptance Rates

Overall Acceptance Rate 55 of 163 submissions, 34%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
373
Total Downloads

Downloads (Last 12 months)23
Downloads (Last 6 weeks)3

Reflects downloads up to 11 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Schoegje THardman LDe Vries APieters T(2024)Improving expert search effectiveness: Comparing ways to rank and present search resultsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638296(56-65)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638296
Peikos GPasi G(2024)A systematic review of multidimensional relevance estimation in information retrievalWIREs Data Mining and Knowledge Discovery10.1002/widm.154114:5Online publication date: 7-May-2024
https://doi.org/10.1002/widm.1541
Van der Sluis FFaure JHomnual S(2024)An empirical exploration of the subjectivity problem of information qualitiesJournal of the Association for Information Science and Technology10.1002/asi.24884Online publication date: 25-Mar-2024
https://doi.org/10.1002/asi.24884
Vakkari P(2023)Assessing Information Search by Task OutcomeSRELS Journal of Information Management10.17821/srels/2023/v60i1/170893(31-37)Online publication date: 27-Mar-2023
https://doi.org/10.17821/srels/2023/v60i1/170893
Li QLiu CHou JWang P(2023)Affective memories and perceived value: motivators and inhibitors of the data search-access processJournal of Documentation10.1108/JD-06-2022-012979:5(1236-1264)Online publication date: 15-Mar-2023
https://doi.org/10.1108/JD-06-2022-0129
Chen JZhang LLu QLiu HChen S(2023)Predicting information usefulness in health information identification from modal behaviorsInformation Processing & Management10.1016/j.ipm.2022.10322060:2(103220)Online publication date: Mar-2023
https://doi.org/10.1016/j.ipm.2022.103220
Brewer RPierce CUpadhyay PPark L(2022)An Empirical Study of Older Adult’s Voice Assistant Use for Health Information SeekingACM Transactions on Interactive Intelligent Systems10.1145/3484507Online publication date: 20-May-2022
https://doi.org/10.1145/3484507
Liu J(2022)Toward Cranfield-inspired reusability assessment in interactive information retrieval evaluationInformation Processing & Management10.1016/j.ipm.2022.10300759:5(103007)Online publication date: Sep-2022
https://doi.org/10.1016/j.ipm.2022.103007
Schoegje Tde Vries APieters T(2022)Adapting a Faceted Search Task Model for the Development of a Domain-Specific Council Information Search EngineElectronic Government10.1007/978-3-031-15086-9_26(402-418)Online publication date: 30-Aug-2022
https://doi.org/10.1007/978-3-031-15086-9_26
Bhattacharya N(2022)A Longitudinal Study to Understand University Students' Searching as a Learning ProcessProceedings of the Association for Information Science and Technology10.1002/pra2.64259:1(404-409)Online publication date: 14-Oct-2022
https://doi.org/10.1002/pra2.642
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents