research-article

Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling

Authors:

David J. Harper,

Iain PirieAuthors Info & Claims

Information Retrieval, Volume 7, Issue 3-4

Pages 265 - 290

https://doi.org/10.1023/B:INRT.0000011207.45988.bb

Published: 01 September 2004 Publication History

Abstract

We present a user-centred, task-oriented, comparative evaluation of two within-document retrieval tools. ProfileSkim computes a relevance profile for a document with respect to a query, and presents the profile as an interactive bar graph. FindSkim provides similar functionality to the web browser “Find” command. A novel simulated work task was devised, where participants are asked to identify (index) relevant pages of an electronic book, given topics from the existing book index. The original book index provides the ground truth, against which the indexing results of the participants can be compared. We confirmed a major hypothesis, namely ProfileSkim proved significantly more efficient than Find-Skim, as measured by time for task. The study indicates that ProfileSkim was as least as effective as FindSkim in identifying relevant pages, as measured by traditional information retrieval measures, and there is some evidence that ProfileSkim is a precision-enhancing tool. Based on qualitative data from questionnaires, we also provide strong evidence to support our conjecture that the participants would be more satisfied when using ProfileSkim than FindSkim. The experimental study confirmed the potential of relevance profiling for improving within-document retrieval. Relevance profiling should prove highly beneficial for users trying to identify relevant information within long documents.

References

[1]

Beaulieu M, Robertson SE, and Rasmussen E Evaluating interactive systems in TREC Journal of the American Society for Information Science 1996 47 1 85-94

[2]

Borlund P and Ingwersen P The development of a method for the evaluation of interactive information retrieval systems Journal of Documentation 1997 53 3 225-250

[3]

procs/ (visited May 15 2003).

[4]

Green TRG (1991) Describing information artefacts with cognitive dimensions and structure maps. In: Diaper D and Hammond NV, Eds., Proceedings of the HCI'91 Conference on People and Computers VI. Cambridge University Press, pp. 297-316.

[5]

Harman D Overview of the First Text REtrieval Conference (TREC-1) 1992 Gaithersburg, Maryland National Institute of Standards and Technology 309-318

[6]

Harper DJ, Coulthard S and Sun Y (2002) A language modelling approach to relevance profiling for document browsing. In: Proceedings of the Joint Conference on Digital Libraries, Oregon, USA, pp. 76-83.

[7]

Harper DJ, Koychev I, and Sun Y Query-based document skimming: A user-centred evaluation of relevance profiling Proceedings of 25th European Conference on Information Retrieval 2003 Berlin Springer-Verlag 377-392

[8]

Hearst MA (1995) TileBars: Visualization of term distribution information in full text information access. In: Proceedings of the ACM SIGCHI Conference on Human Factors in Computing Systems (CHI), Denver, CO, pp. 65-71.

[9]

Hersh W and Over P (2001) TREC-9 interactive track report. In: Proceedings of the Ninth Text Retrieval Conference (TREC-9), Gaithersburg, MD: NIST, pp. 42-50.

[10]

Hersh W, Pentecost J, and Hickam D A task-oriented approach to information retrieval evaluation Journal of the American Society for Information Science 1996 47 1 50-56

[11]

Jansen B, Spink A, and Saracevic T Real life, real users, and real needs: A study and analysis of user queries on the web Information Processing and Management 2000 36 207-227

[12]

Jose J, Furner J and Harper DJ (1998) Spatial querying for image retrieval: A user-oriented evaluation. In: Proceedings of the 21st International ACM-SIGIR Conference on Research and Development in Information Retrieval, ACM Press, pp. 232-240.

[13]

Kaszkiel M and Zobel J (1997) Passage retrieval revisited. In: Proceedings of the Twentieth International ACM-SIGIR Conference on Research and Development in Information Retrieval, Philadelphia, ACM Press, pp. 178-185.

[14]

Pirolli P and Card SK Information foraging Psychological Review 1999 106 643-675

[15]

Ponte J and Croft WB (1998) A language modeling approach to information retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval. ACM Press, pp. 275-281.

[16]

Song F and Croft WB (1999) A general language model for information retrieval. In: Proceedings of the ACM SIGIR Conference on Research and Development in Information Retrieval, ACM Press, pp. 279-280.

[17]

TREC (2002) Interactive track home page. Available at http://www-nlpir.nist.gov/projects/t11i/ (visited May 15 2003).

Cited By

Trippas JSpina DCavedon LJoho HSanderson MShah CBelkin NByström KHuang JScholer F(2018)Informing the Design of Spoken Conversational SearchProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176387(32-41)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1145/3176349.3176387
Loizides FBuchanan GMavri KDobreva MSchmidt B(2015)Theory and practice in visual interfaces for semi-structured document discovery and selectionInformation Services and Use10.3233/ISU-15078335:4(259-271)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.3233/ISU-150783
Wu ZLv YFuxman ALi JWang XGarofalakis MSoboroff ISuel TWang M(2014)Searching Locally-Defined EntitiesProceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management10.1145/2661829.2661954(1499-1508)Online publication date: 3-Nov-2014
https://dl.acm.org/doi/10.1145/2661829.2661954
Show More Cited By

Index Terms

Within-Document Retrieval: A User-Centred Evaluation of Relevance Profiling

Index terms have been assigned to the content through auto-classification.

Recommendations

Non-relevance Feedback for Document Retrieval
KAM '09: Proceedings of the 2009 Second International Symposium on Knowledge Acquisition and Modeling - Volume 02

We need to find documents that relate to human interesting from a large data set of documents. The relevance feedback method needs a set of relevant and non-relevant documents to work usefully. However, the initial retrieved documents, which are ...
A field relevance model for structured document retrieval
ECIR'12: Proceedings of the 34th European conference on Advances in Information Retrieval

Many search applications involve documents with structure or fields. Since query terms often are related to specific structural components, mapping queries to fields and assigning weights to those fields is critical for retrieval effectiveness. Although ...
Adaptive relevance feedback in information retrieval
CIKM '09: Proceedings of the 18th ACM conference on Information and knowledge management

Relevance Feedback has proven very effective for improving retrieval accuracy. A difficult yet important problem in all relevance feedback methods is how to optimally balance the original query and feedback information. In the current feedback methods, ...

Comments

Information & Contributors

Information

Published In

cover image Information Retrieval

Information Retrieval Volume 7, Issue 3-4

Sep 2004

173 pages

ISSN:1386-4564

Issue’s Table of Contents

© Kluwer Academic Publishers 2004.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 September 2004

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Trippas JSpina DCavedon LJoho HSanderson MShah CBelkin NByström KHuang JScholer F(2018)Informing the Design of Spoken Conversational SearchProceedings of the 2018 Conference on Human Information Interaction & Retrieval10.1145/3176349.3176387(32-41)Online publication date: 1-Mar-2018
https://dl.acm.org/doi/10.1145/3176349.3176387
Loizides FBuchanan GMavri KDobreva MSchmidt B(2015)Theory and practice in visual interfaces for semi-structured document discovery and selectionInformation Services and Use10.3233/ISU-15078335:4(259-271)Online publication date: 1-Jan-2015
https://dl.acm.org/doi/10.3233/ISU-150783
Wu ZLv YFuxman ALi JWang XGarofalakis MSoboroff ISuel TWang M(2014)Searching Locally-Defined EntitiesProceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management10.1145/2661829.2661954(1499-1508)Online publication date: 3-Nov-2014
https://dl.acm.org/doi/10.1145/2661829.2661954
Feild HWhite RFu XMackay WBrewster SBødker S(2013)Supporting orientation during search result examinationProceedings of the SIGCHI Conference on Human Factors in Computing Systems10.1145/2470654.2481416(2999-3008)Online publication date: 27-Apr-2013
https://dl.acm.org/doi/10.1145/2470654.2481416
Lassila MPääkkönen TArvola PKekäläinen JJunkkari MKamps JKraaij WFuhr N(2012)Unobtrusive mobile browsing behaviour tracking toolProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362774(278-281)Online publication date: 21-Aug-2012
https://dl.acm.org/doi/10.1145/2362724.2362774
Arvola PVainio JJunkkari MKekäläinen JKamps JKraaij WFuhr N(2012)Model for simulating result document browsing in focused retrievalProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362764(238-241)Online publication date: 21-Aug-2012
https://dl.acm.org/doi/10.1145/2362724.2362764
Schwartz MHash CLiebrock L(2010)Term distribution visualizations with Focus+ContextMultimedia Tools and Applications10.1007/s11042-010-0479-150:3(509-532)Online publication date: 1-Dec-2010
https://dl.acm.org/doi/10.1007/s11042-010-0479-1
Loizides FBuchanan G(2009)An empirical study of user navigation during document triageProceedings of the 13th European conference on Research and advanced technology for digital libraries10.5555/1812799.1812820(138-149)Online publication date: 27-Sep-2009
https://dl.acm.org/doi/10.5555/1812799.1812820
Eslambochilar PBuchanan GLoizides F(2009)Hear it isProceedings of the 13th European conference on Research and advanced technology for digital libraries10.5555/1812799.1812812(75-86)Online publication date: 27-Sep-2009
https://dl.acm.org/doi/10.5555/1812799.1812812
Schwartz MHash CLiebrock LShin SOssowski S(2009)Term distribution visualizations with Focus+ContextProceedings of the 2009 ACM symposium on Applied Computing10.1145/1529282.1529684(1792-1799)Online publication date: 8-Mar-2009
https://dl.acm.org/doi/10.1145/1529282.1529684
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents