article

Methods for Evaluating Interactive Information Retrieval Systems with Users

Author:

Diane KellyAuthors Info & Claims

Foundations and Trends in Information Retrieval, Volume 3, Issue 1—2

Pages 1 - 224

https://doi.org/10.1561/1500000012

Published: 01 January 2009 Publication History

Abstract

This paper provides overview and instruction regarding the evaluation of interactive information retrieval systems with users. The primary goal of this article is to catalog and compile material related to this topic into a single source. This article (1) provides historical background on the development of user-centered approaches to the evaluation of interactive information retrieval systems; (2) describes the major components of interactive information retrieval system evaluation; (3) describes different experimental designs and sampling strategies; (4) presents core instruments and data collection techniques and measures; (5) explains basic data analysis techniques; and (4) reviews and discusses previous studies. This article also discusses validity and reliability issues with respect to both measures and methods, presents background information on research ethics and discusses some ethical issues which are specific to studies of interactive information retrieval (IIR). Finally, this article concludes with a discussion of outstanding challenges and future research directions.

References

[1]

R. P. Abelson, Statistics as Principled Argument . Hillsdale, NJ: Lawrence Erlbaum Publishers, 1995.

[2]

E. Adar, "User 4XXXXX9: Anonymizing query logs," in Proceedings of the Workshop on Query Log Analysis: Social and Technological Challenges, at the 16th International World Wide Web Conference , Banff, Canada, 2007.

[3]

E. Agichtein, E. Brill, and S. Dumais, "Improving web search ranking by incorporating user behavior," in Proceedings of the 29th Annual ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '06) , pp. 3-10, Seattle, WA, 2006.

[4]

J. Allan, "HARD track overview in TREC 2003: High accuracy retrieval from documents," in TREC2003, Proceedings of the 12th Text Retrieval Conference , (E. Voorhees and L. P. Buckland, eds.), Washington, DC: GPO, 2004.

[5]

J. Allan, "HARD track overview in TREC 2005: High accuracy retrieval from documents," in TREC2005, Proceedings of the 14th Text Retrieval Conference , (E. Voorhees and L. P. Buckland, eds.), Washington, DC: GPO, 2006.

[6]

B. Allen, "Information needs: A person-in-situation approach," in Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts , (P. Vakkari, R. Savolainen, and B. Dervin, eds.), pp. 111-122, Tampere, Finland, 1997.

[7]

O. Alonso, D. E. Rose, and B. Stewart, "Crowdsourcing for relevance evaluation," SIGIR Forum , vol. 42, pp. 10-16, 2008.

Digital Library

[8]

R. Amdur, Institutional Review Board Member Handbook . Sudbury, Massachusetts: Jones and Bartlett Publishers, 2003.

[9]

American Psychological Association, Publication Manual of the American Psychological Association . Washington, DC: APA, Fifth ed., 2001.

[10]

E. Amitay, G. C. Murray, and J. Teevan, "Workshop on query log analysis: Social and technological challenges," in Proceedings of the 16th International World Wide Web Conference , Banff, Canada, 2007.

[11]

P. Anick, "Using terminological feedback for web search refinement: A log based study," in Proceedings of the 26th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '03) , pp. 88-95, Toronto, CA, 2003.

[12]

I. Arapakis and J. Jose, "Affective feedback: An investigation of the role of emotions during an information seeking process," in Proceedings of the 31st Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '08) , pp. 395-402, Singapore, Malaysia, 2008.

[13]

E. Babbie, The Practice of Social Research . CA, Wadsworth, 10 ed., 2004.

[14]

P. Bailey, N. Craswell, I. Soboroff, P. Thomas, A. P. de Vries, and E. Yilmaz, "Relevance assessment: Are judges exchangeable and does it matter," in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08) , pp. 667-674, Singapore, Malaysia, 2008.

[15]

M. J. Bates, "Information search tactics," Journal of the American Society for Information Science , vol. 30, pp. 205-214, 1979.

[16]

M. M. Beaulieu, "Interaction in information searching and retrieval," Journal of Documentation , vol. 56, pp. 431-439, 2000.

[17]

M. M. Beaulieu, M. Gatford, X. Huang, S. E. Robertson, S. Walker, and P. Williams, "Okapi at TREC-5," in Proceedings of the 5th Text Retrieval Conference (TREC-5) , (E. M. Voorhees and D. K. Harman, eds.), pp. 143- 165, Washington, DC: GPO, 1997.

[18]

B. Bederson, "Interfaces for staying in the flow," Ubiquity , vol. 5, 2004.

[19]

N. J. Belkin, "Anomalous states of knowledge as a basis for information retrieval," Canadian Journal of Information Science , vol. 5, pp. 133-143, 1980.

[20]

N. J. Belkin, "Helping people find what they don't know," Communications of the ACM , vol. 43, pp. 58-61, 2000.

Digital Library

[21]

N. J. Belkin, A. Cabezas, C. Cool, K. Kim, K. B. Ng, S. Park, R. Pressman, S. Rieh, P. Savage, and I. Xie, "Rutgers interactive track at TREC-5," M. M. Beaulieu, M. Gatford, X. Huang, S. E. Robertson, S. Walker, and P. Williams, "Okapi at TREC-5," in Proceedings of the 5th Text Retrieval Conference (TREC-5) , (E. M. Voorhees and D. K. Harman, eds.), pp. 257- 265, Washington, DC: GPO, 1997, 1997.

[22]

N. J. Belkin, D. Kelly, G. Kim, J.-Y. Kim, H.-J. Lee, G. Muresan, M.-C. Tang, X.-J. Yuan, and C. Cool, "Query length in interactive information retrieval," in Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 205-212, Toronto, Canada, 2003.

[23]

N. J. Belkin and A. Vickery, Interaction in Information Systems: A Review of Research from Document Retrieval to Knowledge-Based Systems . Library and Information Research Report 35: The British Library, University Press, Cambridge, 1985.

[24]

D. J. Bell and I. Ruthven, "Searchers' assessments of task complexity for Web searching," in Proceedings of the 26th Annual International European Conference on Information Retrieval (ECIR 2004) , pp. 57-71, Sunderland, UK, 2004.

[25]

J. L. Bennett, "The user interface in interactive systems," Annual Review of Information Science and Technology , vol. 7, pp. 159-196, 1972.

[26]

J. D. Bernal, "Preliminary analysis of pilot questionnaires on the use of scientific literature," The Royal Society Scientific Information Conference , pp. 589- 637, 1948.

[27]

W. H. Beyer, Handbook of Tables for Probability and Statistics . Cleveland, OH: Chemical Rubber Co. Publishers, 1966.

[28]

M. Bilenko and R. W. White, "Mining the search trails of surfing crowds: Identifying relevant websites from user activity," in Proceedings of the 17th International Conference on the World Wide Web (WWW '08) , pp. 51-60, Beijing, China, 2008.

[29]

A. Blandford, A. Adams, S. Attfield, G. Buchanan, J. Gow, S. Makri, J. Rimmer, and C. Warwick, "The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work," Information Processing and Management , vol. 44, pp. 4-21, 2008.

Digital Library

[30]

C. L. Borgman, "End user behavior on an online information retrieval system: A computer monitoring study," in Proceedings of the 6th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '83) , pp. 162-176, Bethesda, MD, 1983.

[31]

C. L. Borgman, "All users of information retrieval systems are not created equal: An exploration into individual differences," Information Processing and Management , vol. 25, pp. 237-251, 1989.

Digital Library

[32]

P. Borlund, "Experimental components for the evaluation of interactive information retrieval systems," Journal of Documentation , vol. 56, pp. 71-90, 2000.

[33]

P. Borlund, "The concept of relevance in IR," Journal of the American Society for Information Science , vol. 54, pp. 913-925, 2003a.

Digital Library

[34]

P. Borlund, "The IIR evaluation model: A framework for evaluation of interactive information retrieval systems," Information Research , vol. 8, p. 152, 2003b.

[35]

P. Borlund and P. Ingwersen, "The development of a method for evaluating interactive information retrieval systems," Journal of Documentation , vol. 53, pp. 225-250, 1997.

[36]

P. Borlund and P. Ingwersen, "Measure of relative relevance and ranked halflife: Performance indicators for interactive information retrieval," in Proceedings of the 21st ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR '98) , pp. 324-331, Melbourne, Australia, 1998.

[37]

P. Borlund and I. Ruthven, "Introduction to the special issue on evaluating interactive information retrieval systems," Information Processing and Management , vol. 44, pp. 1-3, 2008.

Digital Library

[38]

P. J. Borlund, W. Schneider, M. Lalmas, A. Tombros, J. Feather, D. Kelly, A. P. de Vries, and L. Azzopardi, Proceedings of the 2nd International Symposium on Information Interaction in Context . London, UK, 2008.

Digital Library

[39]

B. R. Boyce, C. T. Meadow, and D. H. Kraft, Measurement in Information Science . London, UK: Academic Press, Inc, 1994.

[40]

J. Bradley, "Methodological issues and practices in qualitative research," Library Quarterly , vol. 63, pp. 431-449, 1993.

[41]

J. Budzik and K. J. Hammond, "User interactions with everyday applications as context for just-in-time information access," in Proceedings of the 5th International Conference on Intelligent User Interfaces (IUI '00) , pp. 44-51, New Orleans, LA, 2000.

[42]

M. Bulmer, Questionnaires V. 1 . Thousand Oaks, CA: Sage Publications, 2004.

[43]

K. Byström, "Information and information sources in tasks of varying complexity," Journal of the American Society for Information Science and Technology , vol. 53, pp. 581-591, 2002.

Digital Library

[44]

K. Byström and P. Hansen, "Conceptual framework for tasks in information studies," Journal of the American Society for Information Science and Technology , vol. 56, pp. 1050-1061, 2005.

[45]

K. Byström and K. Järvelin, "Task complexity affects information seeking and use," Information Processing and Management , vol. 31, pp. 191-213, 1995.

Digital Library

[46]

J. Callan, J. Allan, J. L. A. Clarke, S. Dumais, D. A. Evans, M. Sanderson, and C. Zhai, "Meeting of the MINDS: An information retrieval research agenda," SIGIR Forum , vol. 41, pp. 25-34, 2007.

Digital Library

[47]

D. T. Campbell and J. C. Stanley, Experimental and Quasi-Experimental Designs for Research . Chicago: Rand McNally, 1966.

[48]

R. Capra, "Studying elapsed time and task factors in re-finding electronic information," Personal Information Management, CHI 2008 Workshop , Florence, Italy, 2008.

[49]

D. O. Case, Looking for Information: A Survey of Research on Information Seeking, Needs and Behavior . Lexington, KY: Academic Press, 2002.

[50]

K. Charmaz, "Qualitative interviewing and grounded theory analysis," in Handbook of Interview Research: Context and Method , (J. F. Gubrium and J. A. Holstein, eds.), CA: Sage Publications, 2002.

[51]

E. Chatman, "The impoverished life-world of outsiders," Journal of the American Society for Information Science , vol. 47, pp. 193-206, 1996.

[52]

H. Chen, R. Wigand, and M. Nilan, "Exploring Web users' optimal flow experiences," Information Technology and People , vol. 13, pp. 263-281, 2000.

[53]

J. P. Chin, V. A. Diehl, and K. L. Norman, "Development of an instrument measuring user satisfaction of the human-computer interface," in Proceedings of ACM Human Factors in Computing Systems Conference (CHI 1988) , pp. 213-218, 1988.

[54]

C. W. Cleverdon, "The Cranfield tests on index language devices," in Readings in Information Retrieval , (K. Spark-Jones and P. Willett, eds.), (Reprinted from Aslib Proceedings , pp. 173-192.) San Francisco: Morgan Kaufman Publishers, 1997/1967.

[55]

C. W. Cleverdon, L. Mills, and M. Keen, Factors Determining the Performance of Indexing Systems, vol. 1 - Design . Cranfield, England: Aslib Cranfield Research Project, 1966.

[56]

J. Cohen, Statistical Power Analysis for the Behavioral Sciences . Hillsdale, NJ: Lawrence Earlbaum Associates, Second ed., 1988.

[57]

D. R. Compeau and C. A. Higgins, "Computer self-efficacy: Development of a measure and initial test,"

[58]

C. Cool, "The concept of situation in information science," Annual Review of Information Science and Technology , pp. 5-42, 2001.

[59]

W. S. Cooper, "Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems," American Documentation , vol. 19, pp. 30-41, 1968.

[60]

W. S. Cooper, "On selecting a measure of retrieval effectiveness, part 1: The "subjective" philosophy of evaluation," Journal of the American Society for Information Science , vol. 24, pp. 87-100, 1973.

[61]

P. Cowley, J. Haack, R. Littlefield, and E. Hampson, "Glass Box: Capturing, archiving and retrieving workstation activities," in Proceedings of the 2nd ACM Workshop on Continuous Archival and Retrieval of Personal Experiences (CARPE '05) , pp. 13-18, Santa Barbara, CA, 2006.

[62]

F. Crestani and H. Du, "Written versus spoken queries: A qualitative and quantitative comparative analysis," Journal of the American Society for Information Science and Technology , vol. 57, pp. 881-890, 2006.

Digital Library

[63]

M. Csikszentmihalyi, Finding Flow: The Psychology of Engagement with Everyday Life . New York: Basic Books, 1997.

[64]

M. Csikszentmihalyi and R. Larson, "Validity and reliability of the experiencesampling method," Journal of Nervous and Mental Disease , vol. 175, pp. 526- 536, 1987.

[65]

M. Czerwinski, E. Horvitz, and E. Cutrell, "Subjective duration assessment: An implicit probe for software usability," in Proceedings of IHM-HCI 2001 Conference , pp. 167-170, Lille, France, 2001.

[66]

M. Czerwinski, E. Horvitz, and S. Wilhite, "A diary study of task switching and interruptions," in Proceedings of ACM Human Factors in Computing Systems Conference (CHI '04) , pp. 175-182, Vienna, Austria, 2004.

[67]

N. Dahlbäck, A. Jönsson, and L. Ahrenberg, "Wizard of Oz studies: Why and how," in Proceedings for the 1st International Conference on Intelligent User Interfaces (IUI '93) , pp. 193-200, Orlando, FL, 1993.

[68]

H. Dang, D. Kelly, and J. Lin, "Overview of the TREC 2007 question answering track," in TREC2007, Proceedings of the 16th Text Retrieval Conference , (E. Voorhees and L. P. Buckland, eds.), Washington DC: GPO, 2008.

[69]

F. D. Davis, "Perceived usefulness, perceived ease of use, and user acceptance of information technology," MIS Quarterly , vol. 13, pp. 319-340, 1989.

Digital Library

[70]

M. De Mey, "The cognitive viewpoint: Its development and its scope," in CC77: International Workshop on the Cognitive Viewpoint , (M. De Mey et al., eds.), pp. xvi-xxxi, Ghent, Belgium: University of Ghent Press, 1977.

[71]

S. Debowski, R. Wood, and A. Bandura, "The impact of guided exploration and enactive exploration on self-regulatory mechanisms and information acquisition through electronic enquiry," Journal of Applied Psychology , vol. 86, pp. 1129-1141, 2001.

[72]

S. Dennis, P. Bruza, and R. McArthur, "Web searching: A process-oriented experimental study of three interactive search paradigms," Journal of the American Society for Information Science and Technology , vol. 53, pp. 120- 133, 2002.

Digital Library

[73]

N. K. Denzin and Y. S. Lincoln, Handbook of Qualitative Research . Thousand Oaks, CA: Sage Publications, 2000.

[74]

B. Dervin, "From the mind's eye of the user: The sense-making qualitative-quantitative methodology," in Qualitative Research in Information Management , (R. Glazier, ed.), pp. 61-84, Englewood, CO: Libraries Unlimited, 1992.

[75]

B. Dervin, "Given a context by any other name: Methodological tools for taming the unruly beast," in Information Seeking in Context: Proceedings of an International Conference on Research in Information Needs, Seeking and Use in Different Contexts , pp. 13-38, Tampere, Finland, 1996.

[76]

A. Dillon, "User analysis in HCI: The historical lesson from individual differences research," International Journal of Human-Computer Studies , vol. 45, pp. 619-637, 1996.

Digital Library

[77]

W. D. Dominick and W. D. Penniman, "Automated monitoring to support the analysis and evaluation of information systems," in Proceedings of the 2nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '79) , pp. 2-9, Dallas, TX, 1979.

[78]

P. Dourish, "What we talk about when we talk about context," Personal and Ubiquitous Computing , vol. 8, pp. 19-30, 2004.

Digital Library

[79]

S. Dumais, E. Cutrell, J. J. Cadiz, G. Jancke, R. Sarin, and D. C. Robbins, "Stuff I've Seen: A system for personal information retrieval and re-use," in Proceedings of the ACM Conference on Research and Development in Information Retrieval (SIGIR '03) , pp. 72-79, Toronto, Canada, 2003.

[80]

S. T. Dumais and N. J. Belkin, "The TREC interactive tracks: Putting the user into search," in TREC: Experiment and Evaluation in Information Retrieval , (E. M. Voorhees and D. K. Harman, eds.), pp. 123-153, Cambridge, MA: MIT Press, 2005.

[81]

M. Dunlop, "Time, relevance and interaction modeling for information retrieval," in Proceedings of the 20th ACM Conference on Research and Development in Information Retrieval (SIGIR '97) , pp. 206-213, Philadelphia, PA, 1997.

[82]

M. D. Dunlop, C. W. Johnson, and J. Reid, "Exploring the layers of information retrieval evaluation," Interacting with Computers , vol. 10, pp. 225-236, 1998.

[83]

D. E. Egan, J. R. Remde, L. M. Gomez, T. K. Landauer, J. Eberhardt, and C. C. Lochbaum, "Formative design-evaluation of SuperBook," ACM Transactions on Information Systems , vol. 7, pp. 30-57, 1989.

Digital Library

[84]

M. Eisenberg, "Measuring relevance judgments," Information Processing and Management , vol. 24, pp. 373-389, 1988.

Digital Library

[85]

D. Elgesem, "What is special about the ethical issues in online research?," Ethics and Information Technology , vol. 4, pp. 195-203, 2002.

Digital Library

[86]

D. Elsweiler and I. Ruthven, "Towards task-based personal information management evaluations," in Proceedings of the 30th ACM Conference on Research and Development in Information Retrieval (SIGIR '03) , pp. 22-30, Amsterdam, The Netherlands, 2007.

[87]

K. A. Ericsson and H. A. Simon, Protocol Analysis: Verbal Reports as Data . Cambridge, MA: The MIT Press, Revised ed., 1993.

[88]

C. H. Fenichel, "Online searching: Measures that discriminate among users with different types of experience," Journal of the American Society for Information Science , vol. 32, pp. 23-32, 1981.

[89]

K. D. Fenstermacher and M. Ginsburg, "Client-side monitoring for Web mining," Journal of the American Society for Information Science and Technology , vol. 54, pp. 625-637, 2003.

Digital Library

[90]

R. Fidel, "Online searching styles: A case-study-based model of searching behavior," Journal of the American Society for Information Science , vol. 35, pp. 211-221, 1984.

[91]

R. Fidel, "Qualitative methods in information retrieval research," Library and Information Science Research , vol. 15, pp. 219-247, 1993.

[92]

L. Finkelstein, E. Gabrilovich, Y. Matias, E. Rivlin, Z. Solan, G. Wolfman, and E. Ruppin, "Placing search in context: The concept revisited," Transactions on Information Systems , vol. 20, pp. 116-131, 2002.

Digital Library

[93]

K. E. Fisher, S. Erdelez, and L. E. F. McKenchnie, Theories of Information Behavior . Medford, NJ: Information Today, 2005.

[94]

D. W. Fiske, "Convergent-discriminant validation in measurements and research strategies," in Forms of Validity in Research , (D. Brinbirg and L. H. Kidder, eds.), pp. 77-92, San Francisco: Jossey-Bass, 1982.

[95]

S. T. Fiske, "Mind the gap: In praise of informal sources of formal theory," Personality and Social Psychology Review , vol. 8, pp. 132-137, 2004.

[96]

P. M. Fitts, R. E. Jones, and J. L. Milton, "Eye movements of aircraft pilots during instrument-landing approaches," Aeronautical Engineering Review , vol. 9, pp. 24-29, 1950.

[97]

B. N. Flagg, Formative Evaluation for Educational Technologies . Hillsdale, NJ: Lawrence Erlbaum Associates, 1990.

[98]

S. Flicker, D. Haans, and H. Skinner, "Ethical dilemmas in research on Internet communities," Qualitative Health Research , vol. 14, pp. 124-134, 2004.

[99]

N. Ford, D. Miller, and N. Moss, "The role of individual differences in Internet searching: An empirical study," Journal of the American Society for Information Science and Technology , vol. 52, pp. 1049-1066, 2001.

Digital Library

[100]

N. Ford, D. Miller, and N. Moss, "Web search strategies and human individual differences: Cognitive and demographic factors, Internet attitudes and approaches," Journal of the American Society for Information Science and Technology , vol. 56, pp. 741-756, 2005.

Digital Library

[101]

N. Ford, T. D. Wilson, A. Foster, D. Ellis, and A. Spink, "Information seeking and mediated searching. Part 4: Cognitive styles in information seeking," Journal of the American Society for Information Science and Technology , vol. 53, pp. 728-735, 2002.

Digital Library

[102]

J. Foster, "Collaborative information seeking and retrieval," Annual Review of Information Science and Technology , vol. 40, pp. 329-356, 2006.

Digital Library

[103]

L. Freund and E. Toms, "Revisiting informativeness as a process measure for information interaction," in Proceedings of the Web Information-Seeking and Interaction (WISI) Workshop at the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '07) , pp. 33-36, Amsterdam, The Netherlands, 2007.

[104]

G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais, "The vocabulary problem in human-system communication: An analysis and a solution," Communications of the ACM , vol. 30, pp. 964-971, 1987.

Digital Library

[105]

R. M. Furr and V. R. Bacharach, Psychometrics: An Introduction . Sage Publications, Inc, 2007.

[106]

J. A. Ghani and S. P. Deshpande, "Task characteristics and the experience of optimal flow in Human-Computer interaction," Journal of Psychology , vol. 128, pp. 381-391, 1994.

[107]

J. A. Ghani, R. Supnick, and P. Rooney, "The experience of flow in computer-mediated and in face-to-face groups," in Proceedings of International Conference on Information Systems (ICIS 1991) , pp. 229-237, New York, NY, 1991.

[108]

B. G. Glaser and A. L. Strauss, The Discovery of Grounded Theory: Strategies for Qualitative Research . Chicago: Aldine, 1967.

[109]

A. Göker and H. Myrhaug, "Evaluation of a mobile information system in context," Information Processing and Management , vol. 44, pp. 39-65, 2008.

Digital Library

[110]

F. J. Gravetter and L. B. Wallnau, Statistics for the Behavioral Sciences . Thomson Learning, Fifth ed., 1999.

[111]

C. Grimes, D. Tang, and D. M. Russell, "Query logs alone are not enough," in Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference , Banff, Canada, 2007.

[112]

Z. Guan, S. Lee, E. Cuddihy, and J. Ramey, "The validity of the stimulated retrospective think-aloud method as measured by eye tracking," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems , pp. 1253-1262, Montreal, Canada, 2006.

[113]

J. P. Guilford, Fundamental Statistics in Psychology and Education . New York: McGraw Hill, 1956.

[114]

J. Gwizdka, "Revisiting search task difficulty: Behavioral and individual difference measures," Proceedings of the 71th Annual Meeting of the American Society for Information Science and Technology (ASIS and T) , 2008.

[115]

D. F. Haas and D. H. Kraft, "Experimental and quasi-experimental designs for research in information science," Information Processing and Management , vol. 20, pp. 229-237, 1984.

[116]

K. Halttunen and K. Järvelin, "Assessing learning outcomes in two information retrieval learning environments," Information Processing and Management , vol. 41, pp. 949-972, 2005.

Digital Library

[117]

P. A. Hancock and N. Meshkati, Human Mental Workload . The Netherlands: Elsevier Science Publishers, 1988.

[118]

S. Hansell, AOL removes search data on vase group of Web users, New York Times, Friday, March 14, 2008. Business Section. http://query.nytimes.com/ gst/fullpage.html?res=9504E5D81E3FF93BA3575BC0A9609C8B63, 2006.

[119]

S. Harada, M. Naaman, Y. J. Song, Q.-Y. Wang, and A. Paepcke, "Lost in memories: Interacting with photo collections on PDAs," in Proceedings of the 4th ACM/IEEE-CS Joint Conference on Digital Libraries (JCDL '04) , pp. 325-333, Tuscon, AZ, 2004.

[120]

D. K. Harman, "Introduction to special issue on evaluation issues in information retrieval," Information Processing and Management , vol. 28, pp. 439-440, 1992.

Digital Library

[121]

D. K. Harman, "The TREC test collection," in TREC: Experiment and Evaluation in Information Retrieval , (E. M. Voorhees and D. K. Harman, eds.), pp. 21-52, Cambridge, MA: MIT Press, 2005.

[122]

S. G. Hart and L. E. Staveland, "Development of a NASA-TLX (task load index): Results of empirical and theoretical research," in Human Mental Workload , (P. Hancock and N. Meshkati, eds.), pp. 139-183, The Netherlands: Elsevier Science Publishers, 1988.

[123]

S. P. Harter and C. A. Hert, "Evaluation of information retrieval systems: Approaches, issues and methods," Annual Review of Information Science and Technology , vol. 32, pp. 3-94, 1997.

[124]

M. Hassenzahl, A. Platz, M. Burmester, and K. Lehner, "Hedonic and ergonomic quality aspects determine a software's appeal," Proceedings of the ACM Conference on Human Factors in Computing Systems (CHI '00) , pp. 201-208, 2000.

[125]

D. Hawking, P. Bailey, and N. Craswell, "Efficient and flexible search using text and metadata," CSIRO Mathematical and Information Sciences Tech Report, 2000-83 , available online at http://es.csiro.au/pubs/hawking_ tr00b.pdf, 2000.

[126]

M. D. Heine, "Simulation, and simulation experiments," in Information Retrieval Experiment , (K. Spärck-Jones, ed.), pp. 179-198, London, UK: Butterworths and Co. Ltd, 1981.

[127]

W. Hersh, "Evaluating interactive question answering," in Advances in Open Domain Question Answering , (T. Strzalkowski and S. Harabagiu, eds.), pp. 431-455, Dordrecht, The Netherlands: Springer, 2006.

[128]

W. Hersh and P. Over, "Introduction to a special issue on interactivity at the Text Retrieval Conference (TREC)," Information Processing and Management , vol. 37, pp. 365-367, 2001.

[129]

W. Hersh, A. Turpin, S. Price, B. Chan, D. Kramer, L. Sacherek, and D. Olson, "Do batch and user evaluations give the same results?," in Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '00) , pp. 17-24, Athens, Greece, 2000.

[130]

W. R. Hersh, D. L. Elliot, D. H. Hickam, S. L. Wolf, A. Molnar, and C. Leichtenstein, "Towards new measures of information retrieval evaluation," in Proceedings of the 18th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '95) , pp. 164-170, Seattle, WA, 1995.

[131]

S. Hirsh and J. Dinkelacker, "Seeking information in order to produce information: An empirical study at Hewlett Packard Labs," Journal of the American Society for Information Science and Technology , vol. 55, pp. 807-817, 2004.

Digital Library

[132]

K. Hornbaek, "Current practice in measuring usability: Challenges to usability studies and research," International Journal of Human-Computer Studies , vol. 64, pp. 79-102, 2005.

Digital Library

[133]

K. Hornbæk and E. L.-C. Law, "Meta-analysis of correlations among usability measures," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems , pp. 617-626, San Jose, CA, 2007.

[134]

I. Hsieh-Yee, "Effects of search experience and subject knowledge on the search tactics of novice and experienced searchers," Journal of the American Society for Information Science , vol. 44, pp. 161-174, 1993.

[135]

M. Huang and H. Wang, "The influence of document presentation order and number of documents judged on users' judgments of relevance," Journal of the American Society for Information Science , vol. 55, pp. 970-979, 2004.

Digital Library

[136]

G. Iachello and J. Hong, "End-user privacy in human-computer interaction," Foundations and Trends in Human-Computer Interaction , vol. 1, pp. 1-137, 2007.

Digital Library

[137]

P. Ingwersen, Information Retrieval Interaction . London: Taylor Graham, 1992.

[138]

P. Ingwersen, "Cognitive perspectives of information retrieval interaction: Elements of a cognitive IR theory," Journal of Documentation , vol. 52, pp. 3-50, 1996.

[139]

P. Ingwersen and K. Järvelin, The Turn: Integration of Information Seeking and Retrieval in Context . Dordrecht, The Netherlands: Springer, 2005.

[140]

P. Ingwersen and P. Willett, "An introduction to algorithmic and cognitive approaches for information retrieval," Libri , vol. 45, pp. 160-177, 1995.

[141]

International Standards Office, Ergonomic Requirements for Office Work with Visual Display Terminals (VDTs): Part II, Guidance on Usability (ISO 9241- 11:1998). 1998.

[142]

E. Jacob, "Qualitative research traditions: A review," Review of Educational Research , vol. 57, pp. 1-50, 1987.

[143]

R. J. K. Jacob and K. S. Karn, "Eye tracking in human-Computer interaction and usability research: Ready to deliver the promises (section commentary)," in The Mind's Eye: Cognitive and Applied Aspects of Eye Movement Research , (J. Hyona, R. Radach, and H. Deubel, eds.), pp. 573-605, Amsterdam, Elsevier Science, 2003.

[144]

B. J. Jansen, "Search log analysis: What it is, what's been done, how to do it," Library and Information Science Research , vol. 28, pp. 407-432, 2006.

[145]

B. J. Jansen, R. Ramadoss, M. Zhang, and N. Zang, "Wrapper: An application for evaluating exploratory searching outside of the lab," in Workshop on Evaluating Exploratory Search Systems at the 29th Annual International ACM SIGIR Conference on Research and Development on Information Retrieval (SIGIR '06) , Seattle, WA, 2006.

[146]

B. J. Jansen and A. Spink, "How are we searching the World Wide Web? A comparison of nine search engine transaction logs," Information Processing and Management , vol. 42, pp. 248-263, 2005.

[147]

K. Järvelin, "An analysis of two approaches in information retrieval: From frameworks to study designs," Journal of the American Society for Information Science and Technology , vol. 58, pp. 971-986, 2007.

Digital Library

[148]

K. Järvelin and J. Kekäläinen, "IR evaluation methods for retrieving highly relevant documents," in Proceedings of the 23rd ACM SIGIR Conference on Research and Development of Information Retrieval (SIGIR '00) , pp. 41-48, Athens, Greece, 2000.

[149]

K. Järvelin and J. Kekäläinen, "Cumulated gain-based evaluation of IR techniques," ACM Transactions on Information Systems (TOIS) , vol. 20, pp. 422- 446, 2002.

Digital Library

[150]

K. Järvelin, S. L. Price, L. M. L. Delcambre, and M. L. Nielsen, "Discounted cumulated gain based evaluation of multiple-query IR sessions," in Proceedings of the 30th European Conference on Information Retrieval , Glasgow, Scotland, 2008.

[151]

H. R. Jex, "Measuring mental workload: Problems, process and promises," in Human Mental Workload , (P. Hancock and N. Meshkati, eds.), pp. 5-39, The Netherlands: Elsevier Science Publishers, 1988.

[152]

T. Joachims, L. Granka, B. Pan, H. Hembrooke, and G. Gay, "Accurately interpreting clickthrough data as implicit feedback," in Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '05) , pp. 154-161, Salvador, Brazil, 2005.

[153]

H. Joho, D. Hannah, and J. M. Jose, "Comparing collaborative and independent search in a recall-oriented task," in Proceedings of the 2nd IIiX Symposium on Information Interaction in Context , pp. 89-96, London, UK, 2008.

[154]

H. Joho and J. M. Jose, "Effectiveness of additional representations for the search result presentation on the Web," Information Processing and Management , vol. 44, pp. 226-241, 2008.

Digital Library

[155]

W. Jones and J. Teevan, Personal Information Management . Seattle, WA: University of Washington Press, 2007.

[156]

C.-A. Julien, J. E. Leide, and F. Bouthillier, "Controlled user evaluations of information visualization interfaces for text retrieval: Literature review and meta-analysis," Journal of the American Society for Information Science and Technology , vol. 59, pp. 1012-1024, 2008.

Digital Library

[157]

H. Julien, L. E. F. McKechnie, and S. Hart, "Affective issues in library and information science systems work: A content analysis," Library and Information Science Research , vol. 27, pp. 453-466, 2005.

[158]

M. Käki and A. Aula, "Controlling the complexity in comparing search user interfaces via user studies," Information Processing and Management , vol. 44, pp. 82-91, 2008.

Digital Library

[159]

J. Kalgren and K. Franzen, "Verbosity and interface design," Retrieved on 01 February 2008 at http://www.ling.su.se/staff/franzen/irinterface.html, 1997.

[160]

P. Kantor, G. Kazai, N. Milic-Frayling, and R. Wilkinson, "Proceedings of the 2008 ACM workshop on research advances in large digital book repositories," in Proceedings of the Conference on Information and Knowledge Management (CIKM '08) , Napa, CA, 2008.

[161]

P. B. Kantor, "Evaluation of and feedback in information storage and retrieval systems," Annual Review of Information Science and Technology , vol. 17, pp. 99-120, 1982.

[162]

J. Kekäläinen and K. Järvelin, "Using graded relevance assessments in IR evaluation," Journal of the American Society for Information Science and Technology , vol. 53, pp. 1120-1129, 2002.

Digital Library

[163]

M. Kellar, C. Watters, and M. Shepherd, "A field study characterizing Webbased information-seeking tasks," Journal of the American Society for Information Science and Technology , vol. 58, pp. 999-1018, 2007.

Digital Library

[164]

D. Kelly and X. Fu, "Elicitation of term relevance feedback: An investigation of term source and context," in Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06) , pp. 453-460, Seattle, WA, 2006.

[165]

D. Kelly, J. Harper, and B. Landau, "Questionnaire mode effects in interactive information retrieval experiments," Information Processing and Management , 2008.

[166]

D. Kelly, P. B. Kantor, E. L. Morse, J. Scholtz, and Y. Sun, "User-centered evaluation of interactive question answering systems," in Proceedings of the Workshop on Interactive Question Answering at the Human Language Technology Conference (HLT-NAACL '06) , pp. New York, NY, 2006.

[167]

D. Kelly and J. Lin, "Overview of the TREC 2006 ciQA task," SIGIR Forum , vol. 41, pp. 107-116, 2007.

Digital Library

[168]

D. Kelly, C. Shah, C. R. Sugimoto, E. W. Bailey, R. A. Clemens, A. K. Irvine, N. A. Johnson, W. Ke, S. Oh, A. Poljakova, M. A. Rodriguez, M. G. van Noord, and Y. Zhang, "Effects of performance feedback on users' evaluations of an interactive IR system," in Proceedings of the 2nd Symposium on Information Interaction in Context (IIiX) , pp. 75-82, London, UK, 2008.

[169]

D. Kelly, N. Wacholder, R. Rittman, Y. Sun, P. Kantor, S. Small, and T. Strzalkowski, "Using interview data to identify evaluation criteria for interactive, analytical question answering systems," Journal of the American Society for Information Science and Technology , vol. 58, pp. 1032-1043, 2007.

Digital Library

[170]

J. Kim, "Task as a predictable indicator for information seeking behavior on the Web," Ph.D. Dissertation, Rutgers University, 2006.

[171]

K.-S. Kim and B. Allen, "Cognitive and task influences on Web searching behavior," Journal of the American Society for Information Science and Technology , vol. 53, pp. 109-119, 2001.

Digital Library

[172]

S. Kim and D. Soergel, "Selecting and measuring task characteristics as independent variables," Proceedings of the American Society for Information Science and Technology Conference , vol. 42, 2006.

[173]

D. W. King, "Design and evaluation of information systems," Annual Review of Information Science and Technology , vol. 3, pp. 61-103, 1968.

[174]

C. C. Kuhlthau, Seeking Meaning: A Process Approach to Library and Information Services . Norwood, NJ: Ablex, 1993.

[175]

E. Lagergren and P. Over, "Comparing interactive information retrieval systems across sites: The TREC-6 interactive track matrix experiment," in Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 164-172, Melbourne, Australia, 1998.

[176]

B. Larsen, S. Malik, and A. Tombros, "The interactive track at INEX2005," in INEX 2005 , (N. Fuhr, M. Lalmas, S. Malik, and G. Kazai, eds.), pp. 398-410, Berlin: Springer, 2006.

[177]

Y. Li and N. J. Belkin, "A faceted approach to conceptualizing tasks information seeking," Information Processing and Management , vol. 44, pp. 1822- 1837, 2008.

Digital Library

[178]

R. Likert, "A technique for the measurement of attitudes," Archives of Psychology , vol. 140, pp. 1-55, 1932.

[179]

J. Lin, D. Quan, V. Sinha, K. Bakshi, D. Huynh, B. Katz, and D. R. Karger, "What makes a good answer? The role of context in question answering," in Proceedings of the 9th IFIP TC13 International Conference on Human-Computer Interaction (INTERACT 2003) , (M. Rauterberg, M. Menozzi, and J. Wesson, eds.), Zurich, Switzerland, 2003.

[180]

J. Lin and M. Smucker, "How do users find things with ? Towards automatic utility evaluation with user simulations," in Proceedings of the 31st International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 08) , pp. 19-26, Singapore, Malaysia, 2008.

[181]

S. J. Lin and N. J. Belkin, "Modeling multiple information seeking episodes," in Proceedings of the Annual Conference of the American Society for Information Science (ASIS '00) , pp. 133-146, USA, 2000.

[182]

S. Littlejohn, Theories of Human Communication . Belmont, CA: Wadsworth, 1992.

[183]

I. Lopatovska and H. B. Mokros, "Willingness to pay and experienced utility as measures of affective value of information objects: Users' accounts," Information Processing and Management , vol. 44, pp. 92-104, 2008.

Digital Library

[184]

L. Lorigo, M. Haridasan, H. Brynjarsdottir, L. Xia, T. Joachims, G. Gay, L. Granka, F. Pellacini, and B. Pan, "Eye tracking and online search: Lessons learned and challenges ahead," Journal of the American Society for Information Science and Technology , vol. 59, pp. 1041-1052, 2008.

Digital Library

[185]

L. Lorigo, B. Pan, H. Hembrooke, T. Joachims, L. Granka, and G. Gay, "The influence of task and gender on search and evaluation behavior using Google," Information Processing and Management , vol. 42, pp. 1123-1131, 2006.

Digital Library

[186]

R. M. Losee, "Evaluating retrieval performance given database and query characteristics: Analytical determination of performance surfaces," Journal of the American Society for Information Science , vol. 47, pp. 95-105, 1996.

Digital Library

[187]

H. P. Luhn, "A business intelligence system," in H. P. Luhn: Pioneer of Information Science. Selected Works , (C. K. Shultz, ed.), pp. 132-139, NY: Spartan Books, 1958.

[188]

A. M. Lund, "Measuring usability with the USE Questionnaire," Usability and User Experience , vol. 8, Available online: http://www.stcsig.org/usability/ newsletter/0110_measuring_with_use.html, 2001.

[189]

W. E. Mackay, "Ethics, lies and videotape," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems , pp. 138-145, Denver, CO, 1995.

[190]

G. Marchionini, "Exploratory search: From finding to understanding," Communications of the ACM , vol. 49, pp. 41-46, 2006a.

Digital Library

[191]

G. Marchionini, "Toward human-Computer information retrieval," Bulletin of the American Society for Information Science , online: http://www.asis. org/Bulletin/Jun-06/marchionini.html (retrieved November 7, 2008), 2006b.

[192]

G. Marchionini and G. Crane, "Evaluating hypermedia and learning: Methods and results from the Perseus Project," ACM Transactions on Information Systems , vol. 12, pp. 5-34, 1994.

Digital Library

[193]

J. A. Maxwell, Qualitative Research Design: An Interactive Approach . CA: Sage Publications, 1996.

[194]

D. A. Michel, "What is used during cognitive processing in information retrieval and library searching? Eleven sources of search information," Journal of the American Society for Information Science , vol. 45, pp. 498-514, 1994.

Digital Library

[195]

M. B. Miles and A. M. Huberman, Qualitative Data Analysis: A Sourcebook of New Methods . Newbury Park: Sage, 1984.

[196]

L. I. Millett, B. Friedman, and E. Felten, "Cookies and web browser design: Toward realizing informed consent online," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems , pp. 46-52, Seattle, WA, 2001.

[197]

S. Mizzaro, "Relevance: The whole history," Journal of the American Society for Information Science , vol. 48, pp. 810-832, 1997.

Digital Library

[198]

J. Morahan-Martin, "Males, females, and the Internet," in Psychology and the Internet: Intrapersonal, Interpersonal, and Transpersonal Implications , (J. Gackenback, ed.), pp. 169-197, San Diego: Academic Press, 1998.

[199]

M. R. Morris and E. Horvitz, "SearchTogether: An interface for collaborative Web search," in Proceedings of the 20th Annual ACM Symposium on User Interface Software and Technology (UIST '07) , pp. 3-12, New York, NY, 2007.

[200]

J. L. Myers and A. D. Well, Research Design and Statistical Analysis . Mahway, NJ: Lawrence Erlbaum Associates Inc., Publishers, Second ed., 2003.

[201]

K. A. Neuendorf, The Content Analysis Guidebook . Thousand Oaks, CA: Sage Publications, 2002.

[202]

J. Nielsen, "Usability 101: Introduction to usability," Jakob Nielsen's Alertbox , Retrieved May 05, 2008 from http://www.useit.com/alertbox/20030825.html, 2003.

[203]

J. Nielsen and J. Levy, "Measuring usability: Preference vs performance," Communications of the ACM , vol. 37, pp. 66-75, 1994.

Digital Library

[204]

D. Oard, "Evaluating interactive cross-language information retrieval: Document selection," in Proceedings of the 1st Cross-Language Evaluation Forum , Lisbon, Portugal, 2000.

[205]

H. O'Brien and E. Toms, "What is user engagement? A conceptual framework for defining user engagement with technology," Journal of the American Society for Information Science and Technology , vol. 59, pp. 938-955, 2008.

Digital Library

[206]

H. L. O'Brien, E. G. Toms, E. K. Kelloway, and E. Kelley, "Developing and evaluating a reliable measure of user engagement," in Proceedings of the American Society for Information Science and Technology , Columbus, Ohio, 2008.

[207]

S. L. Payne, The Art of Asking Questions . Princeton, NJ: Princeton University Press, 1951.

[208]

E. J. Pedhazur and L. P. Schmelkin, Measurement, Design and Analysis: An Integrated Approach . Hillsdale, NJ: Lawrence Erlbaum Associates, 1991.

[209]

T. A. Peters, "The history and development of transaction log analysis," Library Hi Tech , vol. 11, pp. 41-66, 1993.

[210]

T. A. Peters, M. Kurth, P. Flaherty, B. Sandore, and N. K. Kaske, "An introduction to the special section on transaction log analysis," Library Hi Tech , vol. 11, pp. 38-40, 1993.

[211]

D. Petrelli, "On the role of user-centred evaluation in the advancement of interactive information retrieval," Information Processing and Management , vol. 44, pp. 22-38, 2008.

Digital Library

[212]

R. W. Picard, Affective Computing . Cambridge, MA: MIT Press, 1997.

[213]

E. M. Pilke, "Flow experiences in information technology use," International Journal of Human-Computer Studies , vol. 61, pp. 347-357, 2004.

Digital Library

[214]

D. J. Pittenger, "Internet research: An opportunity to revisit classic ethical problems in behavioral research," Ethics and Behavior , vol. 13, pp. 45-60, 2003.

[215]

P. M. Podsakoff, S. B. MacKenzie, J.-Y. Lee, and N. P. Podsakoff, "Common method biases in behavioral research: A critical review of the literature and recommended remedies," Journal of Applied Psychology , vol. 88, pp. 879-903,

[216]

R. J. Riding and I. Cheema, "Cognitive styles - An overview and integration," Education Psychology , vol. 11, pp. 193-215, 1991.

[217]

S. Y. Rieh, "Judgment of information quality and cognitive authority in the Web," Journal of the American Society for Information Science and Technology , vol. 53, pp. 145-161, 2002.

Digital Library

[218]

S. E. Robertson, "The probability ranking principle in IR," Journal of Documentation , vol. 33, pp. 294-304, 1977.

Digital Library

[219]

S. E. Robertson, "On GMAP - And other transformations," in Proceedings of the 15th ACM international Conference on information and Knowledge Management (CIKM'06) , pp. 78-83, Arlington, VA, 2006.

[220]

S. E. Robertson, "On the history of evaluation in IR," Journal of Information Science , vol. 34, pp. 439-456, 2008.

Digital Library

[221]

S. E. Robertson and M. M. Hancock-Beaulieu, "On the evaluation of IR systems," Information Processing and Management , vol. 28, pp. 457-466, 1992.

Digital Library

[222]

K. Rodden and X. Fu, "Exploring how mouse movements relate to eye movements on Web search results pages," in Proceedings of the Web Information-Seeking and Interaction (WISI) Workshop at the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '07) , pp. 29-32, Amsterdam, The Netherlands, 2007.

[223]

M. E. Rorvig, "Psychometric measurement and information retrieval," Annual Review of Information Science and Technology , vol. 23, pp. 157-189, 1988.

[224]

I. Ruthven, "Integrating approaches to relevance," in New Directions in Cognitive Information Retrieval , (A. Spink and C. Cole, eds.), pp. 61-80, Netherlands: Springer, 2005.

[225]

I. Ruthven, "Interactive information retrieval," Annual Review of Information Science and Technology , vol. 42, pp. 43-91, 2008.

[226]

I. Ruthven, M. Baillie, and D. Elsweiler, "The relative effects of knowledge, interest and confidence in assessing relevance," Journal of Documentation , vol. 63, pp. 482-504, 2007.

[227]

I. Ruthven, P. Borlund, P. Ingwersen, N. J. Belkin, A. Tombros, and P. Vakkari in Proceedings of the 1st International Conference on Information Interaction in Context , Copenhagen, Denmark, 2006.

Digital Library

[228]

K. J. Ryan, J. V. Brady, R. E. Cooke, D. I. Height, A. R. Jonsen, P. King, K. Lebacqz, D. W. Louisell, D. Seldin, E. Stellar, and R. H. Turtle, "The Belmont Report," Available at http://www.hhs.gov/ohrp/humansubjects/guidance/ belmont.html, 1979.

[229]

G. Salton, "Evaluation problems in interactive information retrieval," Information Storage and Retrieval , vol. 6, pp. 29-44, 1970.

[230]

G. Salton, "The state of retrieval system evaluation," Information Processing and Management , vol. 28, pp. 441-449, 1992.

Digital Library

[231]

T. Saracevic, "Quo vadis test and evaluation," in Proceedings of the Annual Meeting of the American Documentation Institute, 4 , pp. 100-104, New York, NY, 1967.

[232]

T. Saracevic, "Relevance: A review of and a framework for the thinking on the notion in information science," Journal of the American Society for Information Science , vol. 26, pp. 321-343, 1975.

[233]

T. Saracevic, "Evaluation of evaluation in information retrieval," in Proceedings of the 18th Annual ACM SIGIR Conference on Research and Development of Information Retrieval , pp. 138-146, Seattle, WA, 1995.

[234]

T. Saracevic, "The stratified model of information retrieval interaction: Extension and applications," Proceedings of the American Society for Information Science Conference , vol. 34, pp. 313-327, 1997.

[235]

T. Saracevic, "Relevance: A review of the literature and a framework for thinking on the notion in information science. Part II: Nature and manifestations of relevance," Journal of the American Society for Information Science and Technology , vol. 58, pp. 1915-1933, 2007a.

[236]

T. Saracevic, "Relevance: A review of the literature and a framework for thinking on the notion in information science. Part III: Behavior and effects of relevance," Journal of the American Society for Information Science and Technology , vol. 58, pp. 2126-2144, 2007b.

Digital Library

[237]

T. Saracevic and P. B. Kantor, "A study of information seeking and retrieving. II. Users, questions and effectiveness," Journal of the American Society for Information Science , vol. 39, pp. 177-196, 1988a.

[238]

T. Saracevic and P. B. Kantor, "A study of information seeking and retrieving. III. Searchers, searches and overlap," Journal of the American Society for Information Science , vol. 39, pp. 197-216, 1988b.

[239]

T. Saracevic, P. B. Kantor, A. Y. Chamis, and D. Trivison, "A study of information seeking and retrieving: Part 1, background and methodology," Journal of the American Society for Information Science , vol. 39, pp. 161- 176, 1988.

[240]

P. A. Savage-Knepshield and N. J. Belkin, "Interaction in information retrieval: Trends over time," Journal of the American Society for Information Science , vol. 50, pp. 1067-1082, 1999.

Digital Library

[241]

R. Savolainen, "Everyday life information seeking: Approaching information seeking in the context of way of life," Library and Information Science Research , vol. 17, pp. 259-294, 1995.

[242]

R. Siatri, "The evolution of user studies," Libri , vol. 49, pp. 132-141, 1999.

[243]

C. L. Smith and P. B. Kantor, "User adaptation: Good results from poor systems," in Proceedings of the 31st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '08) , pp. 147-154, Singapore, 2008.

[244]

P. Solomon, "Discovering information in context," Annual Review of Information Science and Technology , vol. 36, pp. 229-264, 2002.

[245]

E. Sormunen, "Liberal relevance criteria of TREC: Counting on negligible documents?," in Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR '02) , pp. 324-330, Tampere, Finland, 2002.

[246]

K. Spärck-Jones, Information Retrieval Experiment . London, UK: Butterworths and Co. Ltd, 1981.

[247]

A. Spink, "Multiple search session model of end-user behavior: An exploratory study," Journal of the American Society for Information Science , vol. 47, pp. 603-609, 1996.

Digital Library

[248]

A. Spink, "Study of interactive feedback during mediated information retrieval," Journal of the American Society for Information Science , vol. 48, pp. 382-394, 1997.

[249]

A. Spink and H. Greisdorf, "Regions and levels: Measuring and mapping users' relevance judgments," Journal of the American Society for Information Science and Technology , vol. 52, pp. 161-173, 2001.

Digital Library

[250]

A. Spink and R. M. Losee, "Feedback in information retrieval," Annual Review of Information Science and Technology , vol. 31, pp. 33-78, 1996.

[251]

A. Spink, H. C. Ozmutlu, and S. Ozmutlu, "Multitasking information seeking and searching processes," Journal of the American Society for Information Science and Technology , vol. 53, pp. 639-652, 2002.

Digital Library

[252]

S. R. Stern, "Encountering distressing information in online research: A consideration of legal and ethical responsibilities," New Media and Society , vol. 5, pp. 249-266, 2003.

[253]

A. Strauss and J. Corbin, Basics of Qualitative Research: Grounded Theory Procedures and Techniques . CA: Sage Publications, 1990.

[254]

L. T. Su, "Evaluation measures for interactive information retrieval," Information Processing and Management , vol. 28, pp. 503-516, 1992.

Digital Library

[255]

W. Sugar, "User-centered perspectives of information retrieval research and analysis methods," Annual Review of Information Science and Technology , vol. 30, pp. 77-109, 1995.

[256]

SUMI Questionnaire. (retrieved November 05, 2008), http://www.ucc.ie/hfrg/ questionnaires/sumi/index.html.

[257]

Y. Sun and P. Kantor, "Cross-evaluation: A new model for information system evaluation," Journal of American Society for Information Science and Technology , vol. 56, pp. 614-628, 2006.

[258]

J. Tague, "Informativeness as an ordinal utility function for information retrieval," SIGIR Forum , vol. 21, pp. 10-17, 1987.

Digital Library

[259]

J. Tague and R. Schultz, "Evaluation of the user interface in an information retrieval system: A model," Information Processing and Management , vol. 25, pp. 377-389, 1988.

Digital Library

[260]

J. M. Tague, "The pragmatics of information retrieval experimentation," in Information Retrieval Experiment , (K. S. Jones, ed.), pp. 59-104, London, UK: Butterworths and Co. Ltd, 1981.

[261]

J. M. Tague-Sutcliffe, "The pragmatics of information retrieval experimentation, revisted," Information Processing and Management , vol. 28, pp. 467-490, 1992a.

Digital Library

[262]

J. M. Tague-Sutcliffe, "Measuring the informativeness of a retrieval process," in Proceedings of the 15th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '92) , pp. 23-36, Copenhagen, Denmark, 1992b.

[263]

J. M. Tague-Sutcliffe, Measuring Information: An Information Services Perspective . San Diego, California: Academic Press, 1995.

[264]

J. M. Tague-Sutcliffe, "Some perspectives on the evaluation of information retrieval systems," Journal of the American Society for Information Science , vol. 47, pp. 1-3, 1996.

Digital Library

[265]

R. Tang, M. Shaw, and J. L. Vevea, "Towards the identification of the optimal number of relevance categories," Journal of the American Society for Information Science , vol. 50, pp. 254-264, 1999.

Digital Library

[266]

A. R. Taylor, C. Cool, N. J. Belkin, and W. J. Amadio, "Relationships between categories of relevance criteria and stage in task completion," Information Processing and Management , vol. 43, pp. 1071-1084, 2007.

Digital Library

[267]

R. S. Taylor, "Question negotiation and information seeking in libraries," College and Research Libraries , vol. 29, pp. 178-194, 1968.

[268]

J. Teevan, C. Alvarado, M. S. Ackerman, and D. R. Karger, "The perfect search engine is not enough: A study of orienteering behavior in directed search," in Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (SIGCHI '04) , pp. 415-422, Vienna, Austria, 2004.

[269]

J. Teevan, S. T. Dumais, and E. Horvitz, "Personalizing search via automated analysis of interests and activities," in Proceedings of the 28th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '05) , pp. 449-456, Salvador, Brazil, 2005.

[270]

P. Thomas and D. Hawking, "Evaluation by comparing result sets in context," in Proceedings of the 15th Annual Conference on Information and Knowledge Management (CIKM '06) , pp. 94-101, Arlington, VA, 2006.

[271]

A. Tombros, I. Ruthven, and J. M. Jose, "How users assess Web pages for information seeking," Journal of the American Society for Information Science and Technology , vol. 56, pp. 327-344, 2005.

Digital Library

[272]

E. G. Toms, L. Freund, and C. Li, "WiIRE: The Web interactive information retrieval experimentation system prototype," Information Processing and Management , vol. 40, pp. 655-675, 2004.

Digital Library

[273]

E. G. Toms, H. L. O'Brien, T. Mackenzie, C. Jordan, L. Freund, S. Toze, E. Dawe, and A. MacNutt, "Task effects on interactive search: The query factor," Proceedings of INEX 2007 , pp. 359-372, 2007.

[274]

R. Tourangeau, L. J. Rips, and K. Rasinski, The Psychology of Survey Response . New York, NY: Cambridge University Press, 2000.

[275]

A. Turpin and W. Hersh, "Why batch and user evaluations do not give the same results," in Proceedings of the 24th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '01) , pp. 225-231, New Orleans, LA, 2001.

[276]

A. Turpin and F. Scholer, "User performance versus precision measures for simple search tasks," in Proceedings of the 29th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '06) , pp. 11-18, Seattle, WA, 2006.

[277]

H. Turtle, W. D. Penniman, and T. Hickey, "Data entry/display devices for interactive information retrieval," Annual Review of Information Science and Technology , vol. 16, pp. 55-83, 1981.

[278]

D. J. Urquhart, "The distribution and use of scientific and technical information," The Royal Society Scientific Information Conference , pp. 408-419, 1948.

[279]

P. Vakkari, "Task-based information searching," Annual Review of Information Science and Technology , vol. 37, pp. 413-464, 2003.

[280]

P. Vakkari, "Changes in search tactics and relevance judgments when preparing a research proposal: A summary of the findings of a longitudinal study," Information Retrieval , vol. 4, pp. 295-310, 2004.

[281]

P. Vakkari and N. Hakala, "Changes in relevance criteria and problem stages in task performance," Journal of Documentation , vol. 56, pp. 540-562, 2000.

[282]

P. Vakkari and K. Järvelin, "Explanation in information seeking and retrieval," in New Directions in Cognitive Information Retrieval , (A. Spink and C. Cole, eds.), pp. 113-138, Berlin: Springer, The Information Retrieval Series, 2005.

[283]

P. Vakkari and E. Sormunen, "The influence of relevance levels on the effectiveness of interactive information retrieval," Journal of the American Society for Information Science and Technology , vol. 55, pp. 963-969, 2004.

Digital Library

[284]

A. Veerasamy and N. J. Belkin, "Evaluation of a tool for visualization of information retrieval results," in Proceedings of the 19th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval , pp. 85-92, Zurich, Switzerland, 1996.

[285]

A. Veerasamy and R. Heikes, "Effectiveness of a graphical display of retrieval results," SIGIR Forum , vol. 31, pp. 236-245, 1997.

Digital Library

[286]

M. Viswanathan, Measurement Error and Research Design . Sage Publications, Inc, 2005.

[287]

E. M. Voorhees, "On test collections for adaptive information retrieval," Information Processing and Management , vol. 44, pp. 1879-1885, 2008.

Digital Library

[288]

E. M. Voorhees and D. K. Harman, TREC: Experiment and Evaluation in Information Retrieval . Cambridge, MA: MIT Press, 2005.

[289]

J. B. Walther, "Research ethics in Internet-enabled research: Human subjects issues and methodological myopia," Ethics and Information Technology , vol. 4, pp. 205-216, 2002.

Digital Library

[290]

P. Wang, "Methodologies and methods for user behavior research," Annual Review of Information Science and Technology , vol. 34, pp. 53-99, 1999.

[291]

L. Wen, I. Ruthven, and P. Borlund, "The effects on topic familiarity on online search behaviour and use of relevance criteria," in Proceedings of the 28th European Conference in Information Retrieval (ECIR 2006) , London, UK, 2006.

[292]

R. W. White, M. Bilenko, and S. Cucerzan, "Studying the use of popular destinations to enhance web search interaction," in Proceedings of the 30th Annual ACM International Conference on Research and Development in Information Retrieval (SIGIR '07) , pp. 159-166, Amsterdam, The Netherlands, 2007.

[293]

R. W. White, I. Ruthven, J. M. Jose, and C. J. van Rijsbergen, "Evaluating implicit feedback models using searcher simulations," ACM Transactions on Information Systems , vol. 23, pp. 325-261, 2005.

Digital Library

[294]

B. M. Wildemuth, Applications of Social Research Methods to Questions in Information and Library Science . Libraries Unlimited, (in press).

[295]

B. M. Wildemuth, M. Yang, A. Hughes, R. Gruss, G. Geisler, and G. Marchionini, "Access via features versus access via transcripts: User performance and satisfaction," TREC VID 2003 Notebook Paper , 2003.

[296]

F. Williams, Reasoning with Statistics: How to Read Quantitative Research . Orlando, Florida: Holt, Rinehart and Winston, Inc, Fourth ed., 1992.

[297]

T. D. Wilson, "On user studies and information needs," Journal of Documentation , vol. 37, pp. 3-15, 1981.

[298]

L. Xiong and E. Agichtein, "Towards privacy-preserving query log publishing," in Proceedings of the Workshop on Query Log Analysis: Social and Technology Challenges at the 16th International World Wide Web Conference , Banff, Canada, 2007.

[299]

W. Yuan and C. T. Meadow, "A study of the use of variables in information retrieval user studies," Journal of the American Society for Information Science , vol. 50, pp. 140-150, 1999.

Digital Library

[300]

P. Zhang, L. Plettenberg, J. L. Klavans, D. W. Oard, and D. Soergel, "Taskbased interaction with an integrated multilingual, multimedia information system: A formative evaluation," in Proceedings of the 7th ACM/IEEE Joint Conference on Digital Libraries (JCDL '07) , pp. 117-126, Vancouver, BC, 2007.

[301]

X. Zhang, "Collaborative relevance judgments: A group consensus method for evaluating user search performance," Journal of the American Society for Information Science and Technology , vol. 53, pp. 220-231, 2002.

Digital Library

Cited By

Gaede MBogers THall MKoolen MPetras VLarsen B(2024)The Impact of CHIIR Publications: A Study of Eight Years of CHIIRProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638338(34-44)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638338
Momeni M(2024)Visualization-Enhanced Aggregated Search InterfacesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638336(461-464)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638336
Azzopardi LLiu J(2024)Search under Uncertainty: Cognitive Biases and Heuristics - Tutorial on Modeling Search Interaction using Behavioral EconomicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638297(427-430)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638297
Show More Cited By

Recommendations

Methods for Evaluating Interactive Information Retrieval Systems with Users
Evaluating interactive information retrieval systems: opportunities and challenges
CHI EA '04: CHI '04 Extended Abstracts on Human Factors in Computing Systems

Information retrieval or search plays an important role in a wide range of information management and electronic commerce tasks. In spite of the importance of information retrieval, search systems are often poorly designed from a human computer ...
Evaluating scalability in information retrieval with multigraded relevance
AIRS'06: Proceedings of the Third Asia conference on Information Retrieval Technology

For the user’s point of view, in large environments, it can be desirable to have Information Retrieval Systems (IRS) that retrieve documents according to their relevance levels. Relevance levels have been studied in some previous Information Retrieval (...

Comments

Information & Contributors

Information

Published In

cover image Foundations and Trends in Information Retrieval

Foundations and Trends in Information Retrieval Volume 3, Issue 1—2

January 2009

227 pages

ISSN:1554-0669

EISSN:1554-0677

Issue’s Table of Contents

Publisher

Now Publishers Inc.

Hanover, MA, United States

Publication History

Published: 01 January 2009

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

285
Total Citations
View Citations
3
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Gaede MBogers THall MKoolen MPetras VLarsen B(2024)The Impact of CHIIR Publications: A Study of Eight Years of CHIIRProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638338(34-44)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638338
Momeni M(2024)Visualization-Enhanced Aggregated Search InterfacesProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638336(461-464)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638336
Azzopardi LLiu J(2024)Search under Uncertainty: Cognitive Biases and Heuristics - Tutorial on Modeling Search Interaction using Behavioral EconomicsProceedings of the 2024 Conference on Human Information Interaction and Retrieval10.1145/3627508.3638297(427-430)Online publication date: 10-Mar-2024
https://dl.acm.org/doi/10.1145/3627508.3638297
Liu JAzzopardi LHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Search under Uncertainty: Cognitive Biases and Heuristics: A Tutorial on Testing, Mitigating and Accounting for Cognitive Biases in Search ExperimentsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3661382(3013-3016)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3661382
Schaer PKreutz CBalog KBreuer TFuhr NHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)SIGIR 2024 Workshop on Simulations for Information Access (Sim4IA 2024)Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657991(3058-3061)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657991
Gienapp LScells HDeckers NBevendorff JWang SKiesel JSyed SFröbe MZuccon GStein BHagen MPotthast MHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Evaluating Generative Ad Hoc Information RetrievalProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657849(1916-1929)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657849
Thomas PKazai GCraswell NSpielman SHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)What Matters in a Measure? A Perspective from Large-Scale Search EvaluationProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657845(282-292)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657845
Ji KHettiachchi DSalim FScholer FSpina DHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Characterizing Information Seeking Processes with Multiple Physiological SignalsProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657793(1006-1017)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657793
Łajewska WSpina DTrippas JBalog KHui Yang GWang HHan SHauff CZuccon GZhang Y(2024)Explainability for Transparent Conversational Information-SeekingProceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3626772.3657768(1040-1050)Online publication date: 10-Jul-2024
https://dl.acm.org/doi/10.1145/3626772.3657768
Balog KZhai CChua TNgo CKumar RLauw HKa-Wei Lee R(2024)Tutorial on User Simulation for Evaluating Information Access Systems on the WebCompanion Proceedings of the ACM Web Conference 202410.1145/3589335.3641243(1254-1257)Online publication date: 13-May-2024
https://dl.acm.org/doi/10.1145/3589335.3641243
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents