research-article

Seeding simulated queries with user-study data for personal search evaluation

Authors:

David Elsweiler,

David E. Losada,

José C. Toucedo,

Ronald T. FernandezAuthors Info & Claims

SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

Pages 25 - 34

https://doi.org/10.1145/2009916.2009924

Published: 24 July 2011 Publication History

Abstract

In this paper we perform a lab-based user study (n=21) of email re-finding behaviour, examining how the characteristics of submitted queries change in different situations. A number of logistic regression models are developed on the query data to explore the relationship between user- and contextual- variables and query characteristics including length, field submitted to and use of named entities. We reveal several interesting trends and use the findings to seed a simulated evaluation of various retrieval models. Not only is this an enhancement of existing evaluation methods for Personal Search, but the results show that different models are more effective in different situations, which has implications both for the design of email search tools and for the way algorithms for Personal Search are evaluated.

References

[1]

A. Aula, N. Jhaveri, and M. Käki, Information search and reaccess strategies of experienced web users, Proc. Int. Conf. World Wide Web, 2005, pp. 583--592.

Digital Library

[2]

A. Aula, R. M. Khan, and Z. Guan, How does search behavior change as search becomes more difficult?, Proc. SIGCHI conference on Human factors in computing systems, 2010, pp. 35--44.

Digital Library

[3]

L. Azzopardi, M. de Rijke, and K. Balog, Building simulated queries for known-item topics: an analysis using six european languages, Proc. ACM SIGIR conference on Research and development in information retrieval, 2007, pp. 455--462.

Digital Library

[4]

R. Boardman and M. A. Sasse, "stuff goes into the computer and doesn't come out": a cross-tool study of personal information management, CHI '04: Proc. SIGCHI, 2004, pp. 583--590.

Digital Library

[5]

H. Bruce, W. Jones, and S. Dumais, Keeping and re-finding information on the web: What do people do and what do they need?, ASIST 2004: Proceedings of the 67th ASIST annual meeting, October 2004.

[6]

R. G. Capra and M. A. Perez-Quinones, Re-finding found things: An exploratory study of how users re-find information, Tech. report, Computer Science Dept., Virginia Tech, 2003.

[7]

R. G. Capra and M. A. Perez-Quinones, Using web search engines to find and refind information, Computer 38 (2005), no. 10, 36--42.

Digital Library

[8]

S. Chernov, P. Serdyukov, P. Chirita, G. Demartini, and W. Nejdl, Building a desktop search test-bed, ECIR'07: Proceedings of the 29th European conference on IR research (Berlin, Heidelberg), Springer-Verlag, 2007, pp. 686--690.

Digital Library

[9]

S. Cohen, C. Domshlak, and N. Zwerdling, On ranking techniques for desktop search, ACM Trans. Inf. Syst. 26 (2008), 11:1--11:24.

Digital Library

[10]

E. Cutrell, D. Robbins, S. Dumais, and R. Sarin, Fast, flexible filtering with phlat, Proc. SIGCHI conference on Human Factors in computing systems, 2006, pp. 261--270.

Digital Library

[11]

S. Dumais, E. Cutrell, J. Cadiz, G. Jancke, R. Sarin, and D.C. Robbins, Stuff i've seen: a system for personal information retrieval and re-use, Proc. ACM SIGIR '03:, 2003, pp. 72--79.

Digital Library

[12]

D. Elsweiler, Supporting human memory in personal information management, Ph.D. thesis, Department of Computer and Information Sciences, University of Strathclyde, 2007.

[13]

D. Elsweiler, M. Baillie, and I. Ruthven, Exploring memory in email refinding, ACM Trans. Inf. Syst. 26 (2008), no. 4, 1--36.

Digital Library

[14]

D. Elsweiler, M. Baillie. and I. Ruthven, On understanding the relationship between recollection and refinding, Journal of Digital Information (JoDI) 10 (2009), no. 5.

[15]

D. Elsweiler, M. Baillie, and I. Ruthven, What makes re-finding information diffcult? a study of email re-finding, Proc. ECIR 2011, 2011.

Digital Library

[16]

D. Elsweiler, G. Jones, L. Kelly, and J. Teevan, Workshop on desktop search, SIGIR Forum 44 (2010), 28--34.

Digital Library

[17]

D. Elsweiler and I. Ruthven, Towards task-based personal information management evaluations, Proc. ACM SIGIR conference on Research and development in information retrieval, 2007, pp. 23--30.

Digital Library

[18]

D. Fisher, A. J. Brush, E. Gleave, and M. A. Smith, Revisiting whittaker & sidner's "email overload" ten years later, Proc. Conference on Computer supported cooperative work, 2006, pp. 309--312.

Digital Library

[19]

D. Kelly and J. Teevan, Personal information management, ch. Understanding what works: Evaluating personal information management tools, pp. 190--204, Seattle: University of Washington Press., 2007.

[20]

J. Kim and W. B. Croft, Retrieval experiments using pseudo-desktop collections, CIKM '09: Proceeding of the 18th ACM conference on Information and knowledge management (New York, NY, USA), ACM, 2009, pp. 1297--1306.

Digital Library

[21]

J. Kim and W. B. Croft, Ranking using multiple document types in desktop search, Proceeding of the 33rd international ACM SIGIR conference on Research and development in information retrieval (New York, NY, USA), SIGIR '10, ACM, 2010, pp. 50--57.

Digital Library

[22]

P. McCullagh and J. A. Nelder, phGeneralized linear models (second edition), Chapman and Hall, 1989.

[23]

B. McKenzie and A. Cockburn, An empirical analysis of web page revisitation, Proc. of the HICSS34, 2001.

Digital Library

[24]

P. Ogilvie and J. Callan, Combining document representations for known item search, Proc. ACM SIGIR '03:, 2003, pp. 143--150.

Digital Library

[25]

M. Ringel, E. Cutrell, S. Dumais, and E. Horvitz, Milestones in time: The value of landmarks in retrieving information from personal stores., Proc. INTERACT 2003, 2003, pp. 184--191.

[26]

A. J. Sellen and R. H. R. Harper, The myth of the paperless office, MIT Press, Cambridge, MA, USA, 2003.

Digital Library

[27]

J. Teevan, E. Adar, R. Jones, and M. Potts, Information re-retrieval: Repeat queries in yahoo's logs, Proc. SIGIR '07, 2007.

Digital Library

[28]

J. Teevan, C. Alvarado, M. S. Ackerman, and D. R. Karger, The perfect search engine is not enough: a study of orienteering behavior in directed search, Proc. SIGCHI conference on Human factors in computing systems, 2004, pp. 415--422.

Digital Library

[29]

S. K. Tyler and J. Teevan, Large scale query log analysis of re-finding, Proc. WSDM '10, 2010.

Digital Library

[30]

S. Whittaker and C. Sidner, Email overload: exploring personal information management of email, Proc. SIGCHI conference on Human factors in computing systems, 1996, pp. 276--283.

Digital Library

[31]

Bernard P. Zeigler, Theory of modelling and simulation, Krieger Publishing Co., Inc., Melbourne, FL, USA, 1984.

Digital Library

[32]

C. Zhai and J. Lafferty, A study of smoothing methods for language models applied to information retrieval, ACM Transactions on Information Systems 22 (2004), no. 2, 179--214.

Digital Library

Cited By

Bhargav SSchuth AHauff CChen HDuh WHuang HKato MMothe JPoblete B(2023)When the Music Stops: Tip-of-the-Tongue Retrieval for MusicProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592086(2506-2510)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3592086
Bhargav SSidiropoulos GKanoulas ESelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)'It's on the tip of my tongue'Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498421(48-56)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498421
Kiesel JBahrami AStein BAnand AHagen MAzzopardi LHalvey MRuthven IJoho HMurdock VQvarfordt P(2019)Clarifying False Memories in Voice-based SearchProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298961(331-335)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3295750.3298961
Show More Cited By

Index Terms

Seeding simulated queries with user-study data for personal search evaluation
1. Information systems
  1. Information retrieval

Recommendations

Evaluating user preferences for adaptive reminding
CHI EA '08: CHI '08 Extended Abstracts on Human Factors in Computing Systems

We are developing an adaptive reminding system that tailors its reminders to its users' reminding preferences through real-time interaction and feedback. To determine the potential utility of such a system, we conducted a multi-phase user study, ...
Ambidextrous information search: linking personal and impersonal search routines with individual performance

Organizations invest resources to develop information capabilities, to utilize personal and impersonal information. While the utilization of knowledge is likely to improve organizational performance, it is unclear what the consequences of utilizing ...
Factors in Personal Information Management Evaluation
Advances in Visual Informatics
Abstract
Personal Information Management (PIM) is the domain that involves methods and procedures to study how people keep, manage, maintain and re-find information. Related studies show that re-finding could be a difficult and time-consuming task due to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGIR '11: Proceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval

July 2011

1374 pages

ISBN:9781450307574

DOI:10.1145/2009916

General Chairs:
Wei-Ying Ma
Microsoft Research Asia, China
,
Jian-Yun Nie
University of Montreal, Canada
,
Program Chairs:
Ricardo Baeza-Yates
Yahoo! Research, Spain
,
Tat-Seng Chua
National University of Singapore
,
W. Bruce Croft
University of Massachusetts, Amherst, USA

Copyright © 2011 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGIR: ACM Special Interest Group on Information Retrieval

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 July 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SIGIR '11

Sponsor:

SIGIR

SIGIR '11: The 34th International ACM SIGIR conference on research and development in Information Retrieval

July 24 - 28, 2011

Beijing, China

Acceptance Rates

Overall Acceptance Rate 792 of 3,983 submissions, 20%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
386
Total Downloads

Downloads (Last 12 months)3
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Bhargav SSchuth AHauff CChen HDuh WHuang HKato MMothe JPoblete B(2023)When the Music Stops: Tip-of-the-Tongue Retrieval for MusicProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval10.1145/3539618.3592086(2506-2510)Online publication date: 19-Jul-2023
https://dl.acm.org/doi/10.1145/3539618.3592086
Bhargav SSidiropoulos GKanoulas ESelcuk Candan KLiu HAkoglu LLuna Dong XTang J(2022)'It's on the tip of my tongue'Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining10.1145/3488560.3498421(48-56)Online publication date: 11-Feb-2022
https://dl.acm.org/doi/10.1145/3488560.3498421
Kiesel JBahrami AStein BAnand AHagen MAzzopardi LHalvey MRuthven IJoho HMurdock VQvarfordt P(2019)Clarifying False Memories in Voice-based SearchProceedings of the 2019 Conference on Human Information Interaction and Retrieval10.1145/3295750.3298961(331-335)Online publication date: 8-Mar-2019
https://dl.acm.org/doi/10.1145/3295750.3298961
Vicente-López ECampos LFernández-Luna JHuete JTagua-Jiménez ATur-Vigil C(2015)An automatic methodology to evaluate personalized information retrieval systemsUser Modeling and User-Adapted Interaction10.1007/s11257-014-9148-925:1(1-37)Online publication date: 1-Mar-2015
https://dl.acm.org/doi/10.1007/s11257-014-9148-9
Hagen MWägner DStein B(2015)A Corpus of Realistic Known-Item Topics with Associated Web Pages in the ClueWeb09Advances in Information Retrieval10.1007/978-3-319-16354-3_57(513-525)Online publication date: 2015
https://doi.org/10.1007/978-3-319-16354-3_57
Harvey MElsweiler D(2012)Exploring query patterns in email searchProceedings of the 34th European conference on Advances in Information Retrieval10.1007/978-3-642-28997-2_3(25-36)Online publication date: 1-Apr-2012
https://dl.acm.org/doi/10.1007/978-3-642-28997-2_3

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten