research-article

Reconsideration of the simulated work task situation: a context instrument for evaluation of information retrieval interaction

Authors:

Jesper W. SchneiderAuthors Info & Claims

IIiX '10: Proceedings of the third symposium on Information interaction in context

Pages 155 - 164

https://doi.org/10.1145/1840784.1840808

Published: 18 August 2010 Publication History

Abstract

The present paper reports on the initial study and the preliminary findings of how the concept of simulated work task situation is reported used in the research literature. The overall objective of the study is in a systematic manner to learn how and for what types of evaluations the concept is applied. In particular we are interested to learn whether the recommendations for how to apply simulated work task situations are followed.

The preliminary findings indicate a need for clarifications of the recommendations of how to use simulated work task situations. Particularly with respect to 'realism' of the simulated work task situations, which is emphasised through the need for tailoring of the simulated work task situations towards the group of study participant to ensure the depicted situations are realistic and interesting from the participants' point of view. Likewise it seems that the recommendation to involve the study participants' own information needs (to function as baseline of search interaction) is generally neglected in the reported studies.

References

[1]

Airio, E. (2008). Who benefits from CLIR in web retrieval? Journal of Documentation, 64(5), 760--778.

[2]

Arapakis, I., Jose, J.m. & Gray, P.D. (2008). Affective Feedback: An investigation into the role of emotion in the information seeking process. In: Myaeng, S.H., Oard, D.W. & Sebastiani, F. (eds). Proceedings of the 31st ACM Sigir Conference on Research and Development of Information Retrieval (SIGIR 2008). Singapore, 2008. ACM Press, 395--402.

Digital Library

[3]

Bell, J.D. & Ruthven, I. (2004). Searcher's assessment of task complexity for web searching. In: McDonald, S. & Tait, J. (eds). Proceeding of the 26th European Conference on Information Retrieval (ECIR 2004). 5th-7th April 2004, University of Sunderland, U.K. Berlin/Heidelberg : Springer Verlag, 57-71.

[4]

Bierig, R. & A. Goker. (2006). Time, location and interest: An empirical and user-centred study. In: Ruthven, I., Borlund, P., Ingwersen, P., Belkin, N., Tombros, A. & Vakkari, P. (eds.). Proceedings of the 1st International Symposium on Information Interaction in Context (IIiX 2006). October 18-20 2006, Copenhagen, Denmark. ACM Press, 79--87.

Digital Library

[5]

Bierig, R. (2008). Event and map content personalisation in a mobile and context-aware environment. Robert Gordon University. 2008. {Doctoral Thesis}.

[6]

Blandford, A. et al. (2008). The PRET A Rapporter framework: Evaluating digital libraries from the perspective of information work. Information Processing & Management, 44(1), 4--21.

Digital Library

[7]

Blomgren, L., Vallo, H. & Bystrom, K. (2004). Evaluation of an information system in an information seeking process. In: Heery, R. & Lyon, L. (eds.). Proceedings of the 8th International European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2004). September 12-17 2004, Bath, UK. Berlin/Heidelberg : Springer Verlag, 57--68.

[8]

Borlund, P. & Ingwersen, P. (1997). The development of a method for the evaluation of interactive information retrieval systems. Journal of Documentation, 53(3), 225--250.

[9]

Borlund, P. & Ingwersen, P. (1998). Measures of relative relevance and ranked half-life: performance indicators for interactive IR. In: Croft, B.W, Moffat, A., van Rijsbergen, C.J, Wilkinson, R. & Zobel, J., (eds.). Proceedings of the 21st ACM Sigir Conference on Research and Development of Information Retrieval (SIGIR 1998). Melbourne, Australia, 1998. ACM Press/York Press, 324--331.

Digital Library

[10]

Borlund, P. & Ingwersen, P. (1999). The application of work tasks in connection with the evaluation of interactive information retrieval systems: empirical results. In: Draper, S.W., Dunlop, M.D., Ruthven, I. & van Rijsbergen, C.J., (eds.). Mira '99: Evaluating interactive information retrieval, Glasgow, UK, 14-16 April 1999. British Computer Society, 1--18.

Digital Library

[11]

Borlund, P. (2000). Evaluation of Interactive Information Retrieval Systems. Åbo Akademi University : Åbo Akademi University Press. 2000. {Doctoral thesis, Åbo/Turku}.

[12]

Borlund, P. (2000). Experimental components for the evaluation of interactive information retrieval systems. Journal of Documentation, 56(1), 71--90.

[13]

Borlund, P. (2003). The concept of relevance in IR. Journal of the American Society for Information Science and Technology, 54(10), 913--925.

Digital Library

[14]

Borlund, P. (2003). The IIR evaluation model: A framework for evaluation of interactive information retrieval systems. Information Research, 8(3). Available at: http://informationr.net/ir/8-3/paper152.html

[15]

Borlund, P. (2009). User--centred evaluation of information retrieval systems. In: Goker, A. & Davies, J. (eds.). Information Retrieval: Searching in the 21st Century. John Wiley & Sons, Ldt. ISBN-13: 978-0470027622. 21--37.

[16]

Brajnik, G., Mizzaro, S. & Tasso, C. (2002). Strategic help in user interfaces for information retrieval. Journal of the American Society for Information Science and Technology, 53(5), 343--358.

Digital Library

[17]

Byström, K. & Hansen, P. (2005). Conceptual framework for tasks in information studies. Journal of the American Society for Information Science and Technology, 56(10), 1050--1061.

Digital Library

[18]

Chen, K.H. (2002). Evaluating Chinese text retrieval with multilingual queries. Knowledge Organization, 29(3), 156--170.

[19]

Cole, C. (2001). The concept of situation in Information Science. In: Williams, M.E. (ed.). Annual Review of Information Science and Technology (ARIST). Medford, NJ : Learned Information, INC., 41, 5--42.

[20]

Craven J. & Nietzio, A. (2007). A task-based approach to assessing the accessibility of web sites. Performance Measurement and Metrics, 8(2), 98--109.

[21]

Craven, J. (2003). Access to electronic resources by visually impaired people. Information Research, 8(4). Available at: http://informationr.net/ir/8-4/paper156.html

[22]

Fachry, K.N., Kamps, J. & Zhang, J. (2008). Access to Archival Material in Context. In: Borlund, P., Schneider, J.W., Tombros, A., Lalmas, M., Feather, J., Kelly, D., de Vries, A. & Azzopardi, L. (eds.). Proceeding of the second IIiX Symposium on Information Interaction in Context (IIiX 2008). London, UK, October 2008. ACM Press, 102--109.

Digital Library

[23]

Freund, L. & Toms, E.G. (2006). Enterprise search behavior of software engineers. In: Efthimiadis, E.N., Dumais, S.T., Hawking, D. & Järvelin, K. (eds.). Proceedings of the 29th ACM Sigir Conference on Research and Development of Information Retrieval (SIGIR 2006). Seattle, Washington, USA, August 6-11, 2006. ACM Press, 645--646.

Digital Library

[24]

Freund, L.S. (2008). Exploiting task-document relations in support of information retrieval in the workplace. University of Toronto. 2008. {Doctoral Thesis}.

[25]

Göker, A. & Myrhaug, H. (2008). Evaluation of a mobile information system in context. Information Processing & Management, 44(1), 39--65.

Digital Library

[26]

Haggerty, A., White, R.W. & Jose, J.M. (2003). NewsFlash: Adaptive TV news delivery on the web. In: Nurnberger, A. & Detyniecki, M. (eds.). Proceedings of the 1st International Workshop on Adaptive Multimedia Retrieval, Hamburg (AMR 2003). September 15--16 2003, Germany. Berlin/Heidelberg : Springer Verlag, 72--86.

[27]

Hammer--Aebi, B., Christensen, K.W., Lund, H. & Larsen, B. (2006). Users, structured documents and overlap: Interactive searching of elements and the influence of context on search behavior. In: Ruthven, I., Borlund, P., Ingwersen, P., Belkin, N., Tombros, A. & Vakkari, P. (eds.). Proceedings of the 1st International Symposium on Information Interaction in Context (IIiX 2006). October 18-20 2006, Copenhagen, Denmark. ACM Press, 46--55.

Digital Library

[28]

Harper, D.J., Koychev, I. & Sun, Y.X. (2003). Query-based document skimming: A user-centred evaluation of relevance profiling. In: Sebastiani, F. (ed). Proceeding of the 25th European Conference on Information Retrieval (ECIR 2003). Pisa, Italy. Berlin/Heidelberg : Springer Verlag, 377--392.

Digital Library

[29]

He, D.Q., Brusiloviksy, P., Grady, J., Li, Q. & Ahn, J.W. (2007). How up-to-date should it be? In: Proceedings of ACM International Conference on Web Intelligence (WI 2007), 2-5 November 2007, Silicon Valley, CA, USA. IEEE 2007, 699--705.

Digital Library

[30]

He, D.Q., Brusilovsky, P., Ahn, J., Grady, J., Farzan, R., Peng, Y., Yang, Y. & Rogati, M. (2008). An evaluation of adaptive filtering in the context of realistic task-based information exploration. Information Processing & Management, 44(2), 511--533.

Digital Library

[31]

Hoeber, O. (2007). Exploring Web search results by visually specifying utility functions. In: Proceedings of ACM International Conference on Web Intelligence (WI 2007). 2-5 November 2007, Silicon Valley, CA, USA. IEEE, 650--654.

Digital Library

[32]

Ingwersen, P. (1992). Information retrieval interaction. London: Taylor Graham. X, 246 pp. Available at: http://vip.db.dk/pi/iri/index.htm

Digital Library

[33]

Ingwersen, P. & Järvelin, k. (2005). The turn: Integration of information seeking and retrieval in context. Dordrecht, The Netherlands : Springer Verlag. ISBN:140203850X.

Digital Library

[34]

Jansen, B.J. & McNeese, M.D. (2005). Evaluating the effectiveness of and patterns of interactions with automated searching assistance. Journal of the American Society for Information Science and Technology, 56(14), 1480--1503.

Digital Library

[35]

Järvelin, K. (2007). An analysis of two approaches in information retrieval: From frameworks to study designs. Journal of the American Society for Information Science and Technology, 58(7), 971--986.

Digital Library

[36]

Joho, H. & Jose, J.M. (2005). A comparative study of the effectiveness of search result presentation on the Web. In: Lalmas, M., MacFarlane, A., Rüger, S.M., Tombros, A., Tsikrika, T. & Yavlinsky A. (eds.). Proceedings of the 28th European Conference on IR Research (ECIR 2006). London, UK, April 10-12, 2006. Berlin/Heidelberg : Springer Verlag, 302--313.

Digital Library

[37]

Joho, H. & Jose, J. M. (2006). Slicing and dicing the information space using local context. In: Ruthven, I., Borlund, P., Ingwersen, P., Belkin, N., Tombros, A. & Vakkari, P. (eds.). Proceedings of the 1st International Symposium on Information Interaction in Context (IIiX 2006). October 18--20 2006, Copenhagen, Denmark. ACM Press, 66--74.

Digital Library

[38]

Joho, H. & Jose, J.M. (2008). Effectiveness of additional representations for the search result presentation on the web. Information Processing & Management, 44(1), 226--241.

Digital Library

[39]

Joho, H., Hannah, D. & Jose, J. M. (2008). Comparing collaborative and independent search in a re-call-oriented task. In: Borlund, P., Schneider, J.W., Tombros, A., Lalmas, M., Feather, J., Kelly, D., de Vries, A. & Azzopardi, L. (eds.). Proceeding of the second IIiX Symposium on Information Interaction in Context (IIiX 2008). London, UK, October 2008. ACM Press, 89--96.

Digital Library

[40]

Jose, J.M., Furner, J. & Harper, D.J. (1998). Spatial querying for image retrieval: a user-oriented evaluation. In: Croft, B.W, Moffat, A., van Rijsbergen, C.J, Wilkinson, R. & Zobel, J. (eds.). Proceedings of the 21st ACM Sigir Conference on Research and Development of Information Retrieval (SIGIR 1998). Melbourne, Australia, 1998. ACM Press/York Press, 323--240.

Digital Library

[41]

Karlgren, J. & Hansen, P. (2002). SICS at WLEF 2002: Cross-language relevance assessment and task context. In: Peters, C., Braschler,M., GonzaloJ. & Kluck, M. (eds.). Proceedings of the 3rd Workshop of the Cross-Language Evaluation Forum (CLEF 2002). Rome, Italy, September 19--20, 2002. Berlin/Heidelberg : Springer Verlag, 383--391.

[42]

Kekäläinen, J. (2005). Binary and graded relevance in IR evaluations: Comparison of the effects on ranking of IR systems. Information Processing & Management, 41, 1019--1033.

Digital Library

[43]

Kelly, D. (2009). Methods for evaluating interactive information retrieval systems with users. Foundations and Trends in Information Retrieval, 3(1-2), 1--224.

Digital Library

[44]

Kelly, D. & Belkin N.J. (2004). Display times as implicit feedback: Understanding task effects. In: Sanderson, M., Järvelin, K., Allan, J. & Bruza, P. (eds.). Proceedings of the 27th Conference on Research and Development in Information Retrieval (SIGIR 2004). Sheffield, UK, July 25-29, 2004. ACM, 377--384.

Digital Library

[45]

Kelly, D., Wacholder, N., Rittman, R., Sun, Y., Kantor, P., Small, S. & Strzalkowski, T. (2007). Using interview data to identify evaluation criteria for interactive, analytical question-answering systems. Journal of the American Society for Information Science and Technology, 58(7), 1032--1043.

Digital Library

[46]

Kim, J. (2007). Task as a context of information seeking: An investigation of daily life tasks on the web. Libri, 58, 172--181.

[47]

Kules, B. & Shneiderman, B. (2008). Users can change their web search tactics: Design guidelines for categorized overviews. Information Processing & Management, 44(2), 463--484.

Digital Library

[48]

Langer, L. & Frøkjær, E. (2008). Improving Web search transparency by using a venn diagram interface. In: Tollmar, K. & Jönsson, B. (eds). Proceedings of the 5th Nordic conference on human-computer interaction: Building bridges. Lund, Sweden, October 2008. ACM, 249--256.

Digital Library

[49]

Larsen, B. (2002). Exploiting citation overlaps for information retrieval: Generating a boomerang effect from the network of scientific papers. Scientometrics, 54(2), 155--178.

[50]

Larsen, B., Malik, S. & Tombros, A. (2005). The interactive track at INEX 2005. In: Fuhr, N., Lalmas, M., Malik, S. & Kazai, G. (eds). Proceedings of the 4th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2005). Dagstuhl Castle, Germany, November 28-30, 2005. Berlin/Heidelberg : Springer Verlag, 398--410.

Digital Library

[51]

Malik, S., Klas, C.P., Fuhr, N., Larsen, B. & Tombros, A. (2006). Designing a user interface for interactive retrieval of structured documents: Lessons learned from the INEX interactive track. In: Gonzalo, J. et al (eds). Proceedings of 10th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2006), Alicante, Spain, September 17-22 2006. Berlin/Heidelberg : Springer Verlag, 291--302.

Digital Library

[52]

Malik, S., Larsen, b. & Tombros, A. (2007). Report on the INEX interactive track. ACM SIGIR Forum, June 2007, 41(1), 67--74

Digital Library

[53]

Mizzaro, S. (1998). How many relevances in information retrieval? Interacting with Computers, 10(3), 303--320.

[54]

Nielsen, M.L. (2004). Task-based evaluation of associative thesaurus in real-life environment. In: Schamber, L. & Barry, C.L. (eds.). The Proceedings of the 67th ASIS&T Annual Meeting. Providence, RI., November 12-17 2004. Medford, NJ: Information Today, Inc., 437--447.

[55]

Nielsen, M.L. (2002). The word association method: A gateway to work-task based retrieval. Åbo Akademi University : Åbo Akademi University Press. 2002. {Doctoral thesis, Åbo/Turku}.

[56]

Otterbacher, J., Radev, D. & Kareem, O. (2008). Hierarchical summarization for delivering information to mobile devices. Information Processing & Management, 44(2), 931--947.

Digital Library

[57]

Pansanato, L.T.E. & Fortes, R.P.M. (2007). System description: An orienteering strategy to browse semantically-enhanced educational wiki pages. In: Franconi, E., Kifer, M., May, W. et al. (eds.). Proceedings of the 4th European Semantic Web Conference, Innsbruck, Austria, June 3-7 2007. Berlin/Heidelberg : Springer Verlag, 809--818.

Digital Library

[58]

Papaeconomou, C., Zijlema, A. & Ingwersen, P. (2008). Searchers' relevance judgments and criteria in evaluating web pages in a learning style perspective. In: Borlund, P., Schneider, J.W., Tombros, A., Lalmas, M., Feather, J., Kelly, D., de Vries, A. & Azzopardi, L. (eds.). Proceeding of the second IIiX Symposium on Information Interaction in Context (IIiX 2008). London, UK, October 2008. ACM Press, 123--132.

Digital Library

[59]

Petrelli, D. (2008). On the role of user-centred evaluation in the advancement of interactive information retrieval. Information Processing & Management, 44(1), 22--38.

Digital Library

[60]

Petrelli, D. & Auld, D. (2008). An examination of automatic video retrieval technology on access to the contents of an historical video archive. Program Electronic Library and Information Systems, 42(2), 115--136.

[61]

Petrelli, D., Beaulieu, M., Sanderson, M., Demetriou, G., Herring, P. & Hansen, P. (2004). Observing users, designing clarity: A case study on the user-centered design of a cross-language information retrieval system. Journal of the American Society for Information Science and Technology, 55(10), 923--934.

Digital Library

[62]

Petrelli, D., Demetriou, G., Herring, P., Beaulieu, M. & Sanderson, M. (2002). Exploring the effect of query translation when searching cross-language. In: Peters, C., Braschler,M., GonzaloJ. & Kluck, M. (eds.). Proceedings of the 3rd Workshop of the Cross-Language Evaluation Forum (CLEF 2002). Rome, Italy, September 19-20 2002. Berlin/Heidelberg : Springer Verlag, 430--445.

[63]

Petrelli, D., Levin, S., Beaulieu, M. & Sanderson, M. (2006). Which user interaction for cross-language information retrieval? Design issues and reflections. Journal of the American Society for Information Science and Technology, 57(5), 709--722.

Digital Library

[64]

Pharo, N. (2008). The effect of granularity and order in XML element retrieval. Information Processing & Management, 44(5), 1732--1740.

Digital Library

[65]

Pharo, N. & Nordlie, R. (2005, Jun 04-08). Context matters: An analysis of assessments of XML documents. In: Crestani, F. & Ruthven, I. (eds.). Proceedings of the 5th International Conference on Conceptions of Library and Information Sciences (CoLIS 2005). Glasgow, UK, June 4-8 2005. Berlin/Heidelberg : Springer Verlag, 238--248.

Digital Library

[66]

Price, S.L., Nielsen, M.L., Delcambre, L.M.L. & Vedsted, P. (2007). Semantic components enhance retrieval of domain-specific documents. In: Silva, M.J., Laender, A.H.F., Baeza-Yates, R.A., McGuinness, D.L., Olstad,B., Olsen, Æ.H. & Falcão, A.O. (eds.). Proceedings of the Sixteenth ACM Conference on Information and Knowledge Management (CIKM 2007). Lisbon, Portugal, November 6-10 2007. ACM, 429--438.

Digital Library

[67]

Qu, Y. & Furnas, G.W. (2008). Model-driven formative evaluation of exploratory search: A study under a sensemaking framework. Information Processing & Management, 44(2), 534--555.

Digital Library

[68]

Rodden, K., Basalaj, W., Sinclair, D. & Wood, K. (2001). Does organisation by similarity assist image browsing? In: Proceedings of the SIGCHI on human factors in computing systems. March 31-April 5 2001, Seattle, WA, USA. ACM, 3(1), 190--197.

Digital Library

[69]

Ruthven, I. (2001). Abduction, explanation and relevance feedback. Glasgow University. 2001. {Doctoral thesis}.

[70]

Ruthven, I., Lalmas, M. & van Rijsbergen, K. (2002, Jul 21-25). Ranking expansion terms with partial and ostensive evidence. In: Bruce, H., Fidel, R., Ingwersen, P. & Vakkari, P. (eds.). The proceedings of the 4th International Conference on Conceptions of Library and Information Science (CoLIS 2002). Seattle, Wa, July 21-25 2002. Colerado: Libraries Unlimited, 151--165.

[71]

Ruthven, I., Lalmas, M. & van Rijsbergen, K. (2003). Incorporating user search behavior into relevance feedback. Journal of the American Society for Information Science and Technology, 54(6), 529--549.

Digital Library

[72]

Sagara, Y. (1999). The implications of the non-keyword information on the user's judgment of relevance in information retrieval. {Japanese Journal of} Library and Information Science, 42, 23--46.

[73]

Sihvonen, A. & Vakkari, P. (2004). Subject knowledge improves interactive query expansion assisted by a thesaurus. Journal of Documentation, 60(6), 673--690.

[74]

Skov, M. & Ingwersen, P. (2008). Exploring information seeking behavior in a digital museum context. In: Borlund, P., Schneider, J.W., Tombros, A., Lalmas, M., Feather, J., Kelly, D., de Vries, A. & Azzopardi, L. (eds.). Proceeding of the second IIiX Symposium on Information Interaction in Context (IIiX 2008). London, UK, October 2008. ACM Press, 110--115.

Digital Library

[75]

Skov, M. (2009). The reinvented museum: Exploring information seeking behaviour in a digital museum context. Royal School of Library and Information Science, Copenhagen. 2009. {Doctoral Thesis}.

[76]

Skov, M., Larsen, B. & Ingwersen, P. (2008). Inter and intra-document contexts applied in polyrepresentation for best match IR. Information Processing & Management, 44(5), 1673--1683.

Digital Library

[77]

Spink, A. & Dee, C. (2007). Cognitive shifts related to interactive information retrieval. Online Information Review, 31(6), 845--860.

[78]

Suomela, S. & Kekäläinen, J. (2005). Ontology as a search-tool: A study of real users' query formulation with and without conceptual support. In: Losada, D.E. & Fernández--Luna, J.M. (eds.). Proceedings of the 27th European Conference on IR Research (ECIR 2005). Santiago de Compostela, Spain, March 21-23 2005. Berlin/Heidelberg : Springer Verlag, 315--329.

Digital Library

[79]

Suomela, S. & Kekalainen, J. (2005). User evaluation of ontology as query construction tool. Information Retrieval, 9, 455--475.

Digital Library

[80]

Sutcliffe, A.G., Ennis, M. & Watkinson, S.J. (2000). Empirical studies of end-user information searching. Journal of the American Society for Information Science, 51(13), 1211--1231.

[81]

Szlavik, Z., Tombros, A. & Lalmas, M. (2006) Investigating the use of summarization for interactive XML retrieval. In: Proceedings of the 21st Annual ACM Symposium on Applied Computing (SAC 2006). Dijon, France, April 23-27 2006. ACM, 1068--1072.

Digital Library

[82]

Szlavik, Z., Tombros, A. & Lalmas, M. (2006). The use of summaries in XML retrieval. In: Gonzalo, J. et al (eds). Proceedings of 10th European Conference on Research and Advanced Technology for Digital Libraries (ECDL 2006). Alicante, Spain, September 17-22 2006. Berlin/Heidelberg : Springer Verlag, 75--86.

Digital Library

[83]

Szlavik, Z., Tombros, A., & Lalmas, M. (2007). Feature- and query-based table of contents generation for XML documents. In: Amati, G., Carpineto, C. & Romano, G. (eds.). Proceedings the 29th European Conference on IR Research (ECIR 2007). Rome, Italy, April 2-5 2007. Berlin/Heidelberg : Springer Verlag, 456--467.

Digital Library

[84]

Tombros, A., Larsen, B. & Malik, S. (2004). The interactive track at INEX 2004. In: Fuhr, N., Lalmas, M., Malik, S. & Szlávik, Z. (eds.). Proceedings of the 3rd International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX 2004). Dagstuhl Castle, Germany, December 6-8 2004. Berlin/Heidelberg : Springer Verlag, 410--423.

Digital Library

[85]

Tombros, A., Malik, S. & Larsen, B. (2004). Report on the INEX 2004 interactive track. ACM SIGIR Forum, 39(1), 43--49.

Digital Library

[86]

Tombros, A., Ruthven, I. & Jose, J.M. (2003). Searchers' criteria for assessing web pages. In: Proceedings of the 26th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2003). July 28-August 1 2003, Toronto, Canada. ACM, 385--386.

Digital Library

[87]

Tombros, A., Ruthven, I. & Jose, J.M. (2005). How users assess web pages for information seeking. Journal of the American Society for Information Science and Technology, 56(4), 327--344.

Digital Library

[88]

Toms, E.G., O'Brien, H., Mackenzie, T., Jordan, C., Freund, L., Toze, S., Dawe, E. & MacNutt, A. (2007). Task effects on interactive search: The query factor. In: Fuhr, N., Kamps, J., Lalmas, M. & Trotman, A. (eds.). Proceedings of the 6th International Workshop of the Initiative for the Evaluation of XML Retrieval (INEX), Dagstuhl, Germany. Berlin/Heidelberg : Springer Verlag, 359--372.

Digital Library

[89]

Villa, R., Gildea, N. & Jose, J.M. (2008). Facetbrowser: A user interface for complex search tasks. In: El-Saddik, A., Vuong, S., Griwodz, C., Del Bimbo, A., Candan, K.S. & Jaimes, A. (eds.). Proceedings of the 16th International Conference on Multimedia 2008 (MM '08). Vancouver, British Columbia, Canada, October 26-31 2008. ACM, 489--498.

Digital Library

[90]

Wacholder, N., Kelly, D., Kantor, P., Rittman, R., Sun, Y., Bai, B., Small, S., Yamrom, B. & Strzalkowski, T. (2007). A model for quantitative evaluation of an end-to-end question-answering system. Journal of the American Society for Information Science and Technology, 58(8), 1082--1099.

Digital Library

[91]

Wang, Y.D. & Forgionne, G. (2008). Testing a decision-theoretic approach to the evaluation of information retrieval systems. Journal of Information Science, 34(6), 861--876.

Digital Library

[92]

Wen, L., Ruthven, I. & Borlund, P. (2006). The effects on topic familiarity on online search behavior and use of relevance criteria. In: Lalmas, M., MacFarlane, A., Rüger, S., Tombros, A., Tsikrika, T. & Yavlinsky, A. (eds.). Proceedings of the 28th European Conference on IR Research (ECIR 2006). London, UK, April 2006. Berlin/Heidelberg: Springer Verlag, 456--459.

Digital Library

[93]

White, R.W. & Marchionini, G. (2006). A study of real-time query expansion effectiveness. In: Efthimiadis, E.N., Dumais, S.T., Hawking, D. & Järvelin, K. (eds.). Proceedings of the 29th ACM Sigir Conference on Research and Development of Information Retrieval (SIGIR 2006). Seattle, Washington, USA, August 6-11, 2006. ACM Press, 715--716.

Digital Library

[94]

White, R.W & Ruthven, I. (2006). A study of interface support mechanisms for interactive information retrieval. Journal of the American Society for Information Science and Technology, 57(7), 933--948.

Digital Library

[95]

White, R.W., Jose, J.M. & Ruthven, I. (2001). Query-based web page summarization: A task-oriented evaluation. In: Croft, W.B., Harper, D.J., Kraft, D.H. & Zobel, J. (eds.). Proceedings of the 24th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2001). September 9-13 2001, New Orleans, Louisiana, USA. ACM, 412--413.

Digital Library

[96]

White, R.W., Jose, J.M. & Ruthven, I. (2003). A task-oriented study on the influencing effects of query-biased summarisation in web searching. Information Processing & Management, 39(5), 707--733.

Digital Library

[97]

White, R.W., Jose, J.M. & Ruthven, I. (2005). Using top-ranking sentences to facilitate effective information access. Journal of the American Society for Information Science and Technology, 56(10), 1113--1125.

Digital Library

[98]

White, R.W., Jose, J.M. & Ruthven, I. (2006). An implicit feedback approach for interactive information retrieval. Information Processing & Management, 42(1), 166--190.

Digital Library

[99]

White, R., Bilenko, M. & Cucerzan, S. (2007). Studying the use of popular destinations to enhance web search interaction. In: Kraaij, W., de Vries, A.P., Clarke, C.L.A., Fuhr, N. & Kando, N. (eds.). Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007). Amsterdam, the Netherlands, July 23-27 2007. ACM, 159--166.

Digital Library

[100]

White, R., Bilenko, M. & Cucerzan, S. (2008). Leveraging popular destinations to enhance web search interaction. ACM Transactions on the Web, 2(3), 30 Pp.

Digital Library

[101]

White, R., Ruthven, I. & Jose, J.M. (2001). Web document summarisation: a task-oriented evaluation. In: Tjoa, A.T. & Wagner, R. (eds.). Proceedings of the 12th International Conference on Database and Expert Systems Applications (DEXA 2001). Munich, Germany, September 3-7 2001. Berlin/Heidelberg: Springer Verlag, 551--555.

Digital Library

[102]

White, R.W., Ruthven, I. & Jose, J.M. (2002). Finding relevant documents using top-ranking sentences: An evaluation of two alternative schemes. In: Beaulieu, M., Baeza-Yates, R., Myaeng, S.H. & Järvelin, K. (eds.). Proceedings of the 25th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2002). August 11-15 2002, Tampere, Finland. ACM, 57--64.

Digital Library

[103]

White, R.W., Ruthven, I. & Jose, J.M. (2005). A study of factors affecting the utility of implicit relevance feedback. In: Baeza-Yates, R.A., Ziviani, N., Marchionini, G., Moffat, A. & Tait, J. (eds.). Proceedings of the 28th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2005). Salvador, Brazil, August 15--19 2005. ACM, 35--42.

Digital Library

[104]

Wu, M., Muresan, G. McLean, A., Tang, M., Wilkinson, R., Li. Y., Lee, H. & Belkin, N.J. (2004). Human versus machine in the topic distillation task. In: Sanderson, M., Järvelin, K., Allan, J. & Bruza, P. (eds.). Proceedings of the 27th Conference on Research and Development in Information Retrieval (SIGIR 2004). Sheffield, UK, July 25-29, 2004. ACM, 385--392.

Digital Library

[105]

Xie, I. (2008). Interactive information retrieval in digital environments. Hershey, PA : IGI Pub. ISBN: 978-1-59904-240-4.

Digital Library

[106]

Yuan, X. & Belkin, N.J. (2007). Supporting Multiple Information--seeking strategies in a single system framework. In: Kraaij, W., de Vries, A.P., Clarke, C.L.A., Fuhr, N. & Kando, N. (eds.). Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR 2007). July 23-27 2007, Amsterdam, the Netherlands. ACM, 247--254.

Digital Library

[107]

Yuan, X. (2006). Supporting Multiple Information-seeking strategies in a single system framework. In: Moore, R.C., Bilmes, J.A., Chu-Carroll, J. & Sanderson, M. (eds.). Proceedings of the Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics. June 4-9 2006, New York, USA. The Association for Computational Linguistics, 147--249.

Digital Library

Cited By

Brown TLiu JAizawa AMandl TCarevic ZHinze AMayr PSchaer P(2022)A reference dependence approach to enhancing early prediction of session behavior and satisfactionProceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries10.1145/3529372.3533294(1-5)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3529372.3533294
Aliannejadi MLandoni MHuibers TMurgia EPera MScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Children's Perspective on How Emojis Help Them to Recognise Relevant ResultsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446058(301-305)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446058
Liu J(2021)Deconstructing search tasks in interactive information retrievalInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10252258:3Online publication date: 1-May-2021
https://dl.acm.org/doi/10.1016/j.ipm.2021.102522
Show More Cited By

Index Terms

Reconsideration of the simulated work task situation: a context instrument for evaluation of information retrieval interaction
1. Information systems
  1. Information retrieval

Recommendations

Simulated work tasks: the case of professional users
IIiX '14: Proceedings of the 5th Information Interaction in Context Symposium

This paper investigates simulated work tasks as a tool for information retrieval (IR) evaluation in a work-based, specialized setting. It has been shown that simulated work tasks must be tailored toward the group of study participant to ensure that the ...
Interactive retrieval using simulated versus real work task situations: differences in sub-facets of tasks and interaction performance
ASIST '13: Proceedings of the 76th ASIS&T Annual Meeting: Beyond the Cloud: Rethinking Information Boundaries

Task design in interactive information retrieval (IIR) research, including IIR systems evaluation and IIR behavior research, is critical. Currently, a simple search request and a simulated work task situation design are used most widely, especially the ...
Ontology-based situation awareness

The notions of ''situation'' and ''situation awareness'' have been formulated by many authors in various contexts. In this paper, we present a formalization of situations that is compatible with the interpretation of situation awareness in terms of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

IIiX '10: Proceedings of the third symposium on Information interaction in context

August 2010

408 pages

ISBN:9781450302470

DOI:10.1145/1840784

General Chair:
Nicholas J. Belkin
Rutgers University, USA
,
Program Chair:
Diane Kelly
University of North Carolina at Chapel Hill, USA

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

IEEE
SIGIR: ACM Special Interest Group on Information Retrieval
Microsoft Research: Microsoft Research
Rutgers University: Rutgers University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 18 August 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

IIiX 2010

IIiX 2010: Information Interaction in Context Symposium

August 18 - 21, 2010

New Jersey, New Brunswick, USA

Acceptance Rates

Overall Acceptance Rate 21 of 45 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

30
Total Citations
View Citations
857
Total Downloads

Downloads (Last 12 months)50
Downloads (Last 6 weeks)3

Reflects downloads up to 22 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Brown TLiu JAizawa AMandl TCarevic ZHinze AMayr PSchaer P(2022)A reference dependence approach to enhancing early prediction of session behavior and satisfactionProceedings of the 22nd ACM/IEEE Joint Conference on Digital Libraries10.1145/3529372.3533294(1-5)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3529372.3533294
Aliannejadi MLandoni MHuibers TMurgia EPera MScholer FThomas PElsweiler DJoho HKando NSmith C(2021)Children's Perspective on How Emojis Help Them to Recognise Relevant ResultsProceedings of the 2021 Conference on Human Information Interaction and Retrieval10.1145/3406522.3446058(301-305)Online publication date: 14-Mar-2021
https://dl.acm.org/doi/10.1145/3406522.3446058
Liu J(2021)Deconstructing search tasks in interactive information retrievalInformation Processing and Management: an International Journal10.1016/j.ipm.2021.10252258:3Online publication date: 1-May-2021
https://dl.acm.org/doi/10.1016/j.ipm.2021.102522
Hawking DBillerbeck BThomas PCraswell N(2020)Simulating Information Retrieval Test CollectionsSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S01043ED1V01Y202008ICR07112:2(1-184)Online publication date: 1-Sep-2020
https://doi.org/10.2200/S01043ED1V01Y202008ICR071
Zeynali Tazehkandi MNowkarizi M(2020)Evaluating the effectiveness of Google, Parsijoo, Rismoon, and Yooz to retrieve Persian documentsLibrary Hi Tech10.1108/LHT-11-2019-0229ahead-of-print:ahead-of-printOnline publication date: 8-Mar-2020
https://doi.org/10.1108/LHT-11-2019-0229
Barifah MLandoni MEddakrouri A(2020)Evaluating the user experience in a digital libraryProceedings of the Association for Information Science and Technology10.1002/pra2.28057:1Online publication date: 22-Oct-2020
https://doi.org/10.1002/pra2.280
Mitsui MShah CPiwowarski BChevalier MGaussier EMaarek YNie JScholer F(2019)Bridging GapsProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3331184.3331221(415-424)Online publication date: 18-Jul-2019
https://dl.acm.org/doi/10.1145/3331184.3331221
Zhu YKim MYan E(2019)Evaluating interactive bibliographic information retrieval systems: A user‐centered approachProceedings of the Association for Information Science and Technology10.1002/pra2.2018.1450550106855:1(628-637)Online publication date: Feb-2019
https://doi.org/10.1002/pra2.2018.14505501068
Liu YLiu CBierig RSong DLiu TSun LBruza PMelucci MSebastiani FYang G(2018)User Interactions with Search SystemsProceedings of the 2018 ACM SIGIR International Conference on Theory of Information Retrieval10.1145/3234944.3234981(3-4)Online publication date: 10-Sep-2018
https://dl.acm.org/doi/10.1145/3234944.3234981
Kayal ABrinkman WNeerincx MRiemsdijk M(2018)Automatic Resolution of Normative Conflicts in Supportive Technology Based on User ValuesACM Transactions on Internet Technology10.1145/315837118:4(1-21)Online publication date: 25-May-2018
https://dl.acm.org/doi/10.1145/3158371
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten