research-article

Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework

Authors:

Jaime Arguello,

Ashlee Edwards,

Wan-ching WuAuthors Info & Claims

ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

Pages 101 - 110

https://doi.org/10.1145/2808194.2809465

Published: 27 September 2015 Publication History

Abstract

One of the most challenging aspects of designing interactive information retrieval (IIR) experiments with users is the development of search tasks. We describe an evaluation of 20 search tasks that were designed for use in IIR experiments and developed using a cognitive complexity framework from educational theory. The search tasks represent five levels of cognitive complexity and four topical domains. The tasks were evaluated in the context of a laboratory IIR experiment with 48 participants. Behavioral and self-report data were used to characterize and understand differences among tasks. Results showed more cognitively complex tasks required significantly more search activity from participants (e.g., more queries, clicks, and time to complete). However, participants did not evaluate more cognitively complex tasks as more difficult and were equally satisfied with their performances across tasks. Our work makes four contributions: (1) it adds to what is known about the relationship among task, search behaviors and user experience; (2) it presents a framework for task creation and evaluation; (3) it provides tasks and questionnaires that can be reused by others and (4) it raises questions about findings and assumptions of many recent studies that only use behavioral signals from search logs as evidence for task difficulty and searcher satisfaction, as many of our results directly contradict these findings.

References

[1]

Ageev, M., Guo, Q., Lagun, D. & Agichtein, E. (2011). Find it if you can: A game for modeling different types of web search success using interaction data. Proc. of SIGIR, 345--354.

Digital Library

[2]

Allan, J., Croft, B., Moffat, A. & Sanderson, M. (Eds). (2012). Frontiers, challenges and opportunities for Information retrieval: Report from SWIRL 2012. SIGIR Forum, 46(1), 2--32.

Digital Library

[3]

Anderson, L. W. & Krathwohl, D. A. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom's taxonomy of educational objectives. New York: Longman.

[4]

Arguello, J. (2014). Predicting search task difficulty. Proc. ECIR, 88--99.

Digital Library

[5]

Arguello, J., Wu, W.C., Kelly, D., & Edwards, A. (2012). Task complexity, vertical display and user interaction in aggregated search. Proc. of SIGIR, 435--444.

Digital Library

[6]

Aula, A., Khan, R. M. & Guan, Z. (2010). How does search behavior change as search becomes more difficult? Proc. of SIGCHI Conference, 35--44.

Digital Library

[7]

Bailey, P., Moffat, A., Scholer, F., & Thomas, P. (2015). User Variability and IR System Evaluation. Proc. of SIGIR.

Digital Library

[8]

Bell, D. J. & Ruthven, I. (2004). Searcher's assessments of task complexity for web searching. Proc. of ECIR, 57--71.

[9]

Belkin, N. J., Dumais, S. Kando, N. & Sanderson, M. (2012, October). Whole Session Evaluation of Interactive Information Retrieval Systems. National Institute of Informatics Shonan Meeting, Shonan Village Center, Japan.

[10]

Borlund, P. (2003). The IIR evaluation model: A framework for the evaluation of interactive information retrieval systems. Information Research, 8(3), paper 152.

[11]

Brennan, K., Kelly, D., & Arguello, J. (2014). The effect of cognitive abilities on information search for tasks of varying levels of complexity. Proc. of IIiX, 165--174.

Digital Library

[12]

Broder, A. (2002). A taxonomy of web search. SIGIR Forum, 36(2), 3--10.

Digital Library

[13]

Byström, K. & Hansen, P. (2005). Conceptual framework for tasks in information studies. JASIST, 56(10), 1050--1061.

[14]

Byström, K. & Järvelin, K. (1995). Task complexity affects information seeking and use. IP&M, 31, 191--213.

Digital Library

[15]

Campbell, D. J. (1988). Task complexity: A review and analysis. Academy of Management Review, 13, 40--52.

[16]

Crescenzi, A., Capra, R. & Arguello, J. (2013). Time Pressure, User Satisfaction and Task Difficulty. Proc. of ASIST Conference.

Digital Library

[17]

Diriye, A., White, R. W., Buscher, G., & Dumais, S. T. (2012). Leaving so soon? Understanding and predicting web search abandonment rationales. Proc. of CIKM, 1025--1034.

Digital Library

[18]

Dumais, S. T., & Belkin, N. J. (2005). The TREC Interactive Tracks: Putting the user into search. In E. M. Voorhees & D. K. Harman (Eds.) TREC: Experiment and Evaluation in Information Retrieval (pp. 123--153), Cambridge, MA: MIT Press.

[19]

Gwizdka, J. & Spence, I. (2006). What can searching behavior tell us about the difficulty of information tasks? A study of Web navigation. Proc. of ASIST, 1--22.

[20]

Hassan, A., White, R. W., Dumais, S. T., & Wang, Y. M. (2014). Struggling or exploring?: Disambiguating long search sessions. Proc. of WSDM, 53--62.

Digital Library

[21]

Jansen, B. J., Booth, D. & Smith, B. (2009). Using the taxonomy of cognitive learning to model online searching. IP&M, 45, 643--663.

Digital Library

[22]

Jiang, J., He, D. & Allan, J. (2014). Searching, browsing and clicking in a search session: Changes in user behavior by task and over time. Proc. of SIGIR, 607--616.

Digital Library

[23]

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2).

Digital Library

[24]

Kelly, D., Arguello, J., & Capra, R. (2013). NSF workshop on task-based information search systems. SIGIR Forum, 47(2).

Digital Library

[25]

Kelly, D., Dumais, S., & Pedersen, J. (2009). Evaluation challenges and directions for information seeking support systems. IEEE Computer, 42(3), 60--66.

Digital Library

[26]

Kim, J. (2006). Task difficulty as a predictor and indicator of web searching interaction. Proc. of CHI (Extended Abstracts), 959--964.

Digital Library

[27]

Li, Y. & Belkin, N. J. (2008). A faceted approach to conceptualizing tasks in information seeking. IP&M, 44, 1822--1837.

Digital Library

[28]

Li, Y. & Belkin, N. J. (2010). An exploration of the relationship between work task and interactive information search behavior. JASIST, 61(9), 1771--1789.

Digital Library

[29]

Liu, J., Liu, C., Cole, M., Belkin, N. J., & Zhang, X. (2012). Exploring and predicting search task difficulty. Proc. of CIKM, 1313--1322.

Digital Library

[30]

Liu, J., Kim, C. S. & Creel, C. (2013). Why do users feel search task difficult? Proc. of ASIST.

Digital Library

[31]

Liu, Q., Agichtein, E., Dror, G., Maarek, Y. & Szpektor, I. (2012). When web search fails, searchers become askers: Understanding the transition. Proc. of SIGIR, 801--810.

Digital Library

[32]

Moffat, A., Thomas, P., & Scholer, F. (2013). Users versus models: What observation tells us about effectiveness metrics. Proc. of CIKM, 659--668.

Digital Library

[33]

Robertson, S. (2008). On the history of evaluation in IR. Journal of Documentation, 34(4), 439--456.

Digital Library

[34]

Singer, G., Norbisrath, U. & Lewandowski, D. (2012). Ordinary search engine users assessing difficulty, effort, and outcome for simple and complex search tasks. Proc. of IIiX, 110--119.

Digital Library

[35]

Smith, C. L. (2008). Searcher adaptation: A response to topic difficulty. Proc. of the ASIST Conference.

[36]

Toms, E. (2011). Task-based information searching and retrieval. In Ruthven, I., & Kelly, D. (Eds.) Interactive Information-seeking, Behaviour and Retrieval (pp. 43--59).

[37]

Toms, E., O'Brien, H. L., MacKenzie, T., Jordan, C., Freund, L., Toze, S., Dawe, E., & MacNutt, A. (2007). Task effects on interactive search: The query factor. Proc. of INEX, 359--372.

[38]

Vakkari, P. (2003). Task-based information searching. ARIST, 37, 413--464.

[39]

White, R.W. & Dumais, S. T. (2009). Characterizing and predicting search engine switching behavior. Proc. of CIKM, 87--96.

Digital Library

[40]

White, R. W. & Roth, R.A. (2009). Exploratory search: Beyond the query-response paradigm. Morgan & Claypool.

Digital Library

[41]

Whittaker, S., Terveen, L., & Nardi, B. (2000). Let's stop pushing the envelope and start addressing it: A reference task agenda for HCI. Human Computer Interaction, 15, 75--106.

Digital Library

[42]

Wildemuth, B. W., Freund, L. & Toms, E. G. (2014). Untangling search task complexity and difficulty in the context of interactive information retrieval studies. Journal of Documentation, 70(6), 1118--1140.

[43]

Wu, W.C., Kelly, D., Edwards, A., & Arguello, J. (2012). Grannies, tanning beds, tattoos and NASCAR: Evaluation of search tasks with varying levels of cognitive complexity. Proc. of IIiX, 254--257.

Digital Library

Cited By

Urgo KArguello J(2024)The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning ProcessesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638348(278-290)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638348
Li YCapra R(2024)Stopped yet Completed: Exploring the Relationships between Session-stopping Reasons, Information Types, and Cognitive Activities in Cross-Session SearchesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638304(119-129)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638304
Ghosh SGhosh SShah CKlein MBen-David AJäschke RKelly M(2024)Toward Connecting Speech Acts and Search Actions in Conversational Search TasksProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00027(119-131)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00027
Show More Cited By

Index Terms

Development and Evaluation of Search Tasks for IIR Experiments using a Cognitive Complexity Framework
1. Information systems
  1. Information retrieval
    1. Information retrieval query processing

Recommendations

Grannies, tanning beds, tattoos and NASCAR: evaluation of search tasks with varying levels of cognitive complexity
IIIX '12: Proceedings of the 4th Information Interaction in Context Symposium

One of the most challenging aspects of designing an interactive information retrieval (IIR) study is the development of search tasks. In this paper, we present preliminary results of a study designed to evaluate a set of search tasks that were developed ...
An examination of multisession web tasks

Today, people perform many types of tasks on the web, including those that require multiple web sessions. In this article, we build on research about web tasks and present an in-depth evaluation of the types of tasks people perform on the web over ...
An exploration of the relationships between work task and interactive information search behavior

This study explores the relationships between work task and interactive information search behavior. Work task was conceptualized based on a faceted classification of task. An experiment was conducted with six work-task types and simulated work-task ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICTIR '15: Proceedings of the 2015 International Conference on The Theory of Information Retrieval

September 2015

402 pages

ISBN:9781450338332

DOI:10.1145/2808194

General Chairs:
James Allan
University of Massachusetts Amherst, USA
,
Bruce Croft
University of Massachusetts Amherst, USA
,
Program Chairs:
Arjen de Vries
CWI Amsterdam, The Netherlands
,
Chengxiang Zhai
University of Illinois at Urbana-Champaign, USA

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 September 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICTIR '15

Sponsor:

SIGIR

ICTIR '15: ACM SIGIR International Conference on the Theory of Information Retrieval

September 27 - 30, 2015

Massachusetts, Northampton, USA

Acceptance Rates

ICTIR '15 Paper Acceptance Rate 29 of 57 submissions, 51%;

Overall Acceptance Rate 235 of 527 submissions, 45%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

97
Total Citations
View Citations
834
Total Downloads

Downloads (Last 12 months)70
Downloads (Last 6 weeks)6

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Urgo KArguello J(2024)The Effects of Goal-setting on Learning Outcomes and Self-Regulated Learning ProcessesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638348(278-290)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638348
Li YCapra R(2024)Stopped yet Completed: Exploring the Relationships between Session-stopping Reasons, Information Types, and Cognitive Activities in Cross-Session SearchesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638304(119-129)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638304
Ghosh SGhosh SShah CKlein MBen-David AJäschke RKelly M(2024)Toward Connecting Speech Acts and Search Actions in Conversational Search TasksProceedings of the 2023 ACM/IEEE Joint Conference on Digital Libraries10.1109/JCDL57899.2023.00027(119-131)Online publication date: 26-Jun-2024
https://dl.acm.org/doi/10.1109/JCDL57899.2023.00027
Jung YLiu J(2024)Toward a conceptual framework characterizing the interplay of interest development, information search, and knowledge construction (ISK) in Children’s learningAslib Journal of Information Management10.1108/AJIM-01-2024-0041Online publication date: 24-Jun-2024
https://doi.org/10.1108/AJIM-01-2024-0041
Wang BLiu J(2024)Understanding users' dynamic perceptions of search gain and cost in sessions: An expectation confirmation modelJournal of the Association for Information Science and Technology10.1002/asi.24935Online publication date: 17-Jun-2024
https://doi.org/10.1002/asi.24935
Choi BCasteel SArguello JCapra R(2023)Better Understanding Procedural Search Tasks: Perceptions, Behaviors, and ChallengesACM Transactions on Information Systems10.1145/363000442:3(1-32)Online publication date: 29-Dec-2023
https://dl.acm.org/doi/10.1145/3630004
Shah CWhite RThomas PMitra BSarkar SBelkin N(2023)Taking Search to TaskProceedings of the 2023 Conference on Human Information Interaction and Retrieval10.1145/3576840.3578288(1-13)Online publication date: 19-Mar-2023
https://dl.acm.org/doi/10.1145/3576840.3578288
Zou JAliannejadi MKanoulas EPera MLiu Y(2023)Users Meet Clarifying Questions: Toward a Better Understanding of User Interactions for Search ClarificationACM Transactions on Information Systems10.1145/352411041:1(1-25)Online publication date: 9-Jan-2023
https://dl.acm.org/doi/10.1145/3524110
Dosso CTamine LPaubel PChevalier A(2023)Navigational and thematic exploration–exploitation trade-offs during web search: effects of prior domain knowledge, search contexts and strategies on search outcomeBehaviour & Information Technology10.1080/0144929X.2023.224251443:10(2232-2258)Online publication date: 10-Aug-2023
https://doi.org/10.1080/0144929X.2023.2242514
Liu JLiu J(2023)Formally Modeling Users in Information RetrievalA Behavioral Economics Approach to Interactive Information Retrieval10.1007/978-3-031-23229-9_2(23-64)Online publication date: 18-Feb-2023
https://doi.org/10.1007/978-3-031-23229-9_2
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents