Article

Providing consistent and exhaustive relevance assessments for XML retrieval evaluation

Authors:

Benjamin Piwowarski,

Mounia LalmasAuthors Info & Claims

CIKM '04: Proceedings of the thirteenth ACM international conference on Information and knowledge management

Pages 361 - 370

https://doi.org/10.1145/1031171.1031246

Published: 13 November 2004 Publication History

Get Access

Abstract

Comparing retrieval approaches requires test collections, which consist of documents, queries and relevance assessments. Obtaining consistent and exhaustive relevance assessments is crucial for the appropriate comparison of retrieval approaches. Whereas the evaluation methodology for flat text retrieval approaches is well established, the evaluation of XML retrieval approaches is a research issue. This is because XML documents are composed of nested components that cannot be considered independent in terms of relevance. This paper describes the methodology adopted in INEX (the INitiative for the Evaluation of XML Retrieval) to ensure consistent and exhaustive relevance assessments.

References

[1]

R. Baeza-Yates, N. Fuhr, and Y. S. Maarek, editors. ACM SIGIR 2002 Workshop on XML, Aug. 2002.

Google Scholar

[2]

H. M. Blanken, T. Grabs, H.-J. S. andRalf Schenkel, and G. Weikum, editors. Intelligent Search on XML Data, Applications, Languages, Models, Implementations, and Benchmarks, volume 2818 of Lecture Notes in Computer Science. Springer, 2003.

Google Scholar

[3]

D. Carmel, Y. Maarek, and A. Soffer, editors. ACM SIGIR 2000 Workshop on XML, July 2000.

Google Scholar

[4]

Y. Chiaramella. Browsing and Querying: two complementary approaches for Multimedia Information Retrieval. In HIM'97 International Conference, Dortmund, Germany, 1997.

Google Scholar

[5]

N. Fuhr, S. Malik, and M. Lalmas. Overview of the initiative for the evaluation of xml retrieval (inex) 2003. In Proceedings of the Second INEX Workshop, March 2004.

Google Scholar

[6]

G. Kazai. Report of the inex 2003 metrics working group. In N. Fuhr, M. Lalmas, and S. Malik, editors, Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML retrieval (INEX), Dagstuhl, germany, December 2003, pages 184--190, April 2004.

Google Scholar

[7]

G. Kazai, S. Masood, and M. Lalmas. A study of the assessment of relevance for the INEX'02 test collection. In S. McDonald and J. Tait, editors, Advances in Information Retrieval, 26th European Conference on IR Research, ECIR 2004, volume 2997 of Lecture Notes in Computer Science, Sunderland, UK, Apr. 2004. Springer.

Google Scholar

[8]

J. Kekäläinen and K. Järvelin. Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science (JASIS), 53(13):1120--1129, 2002.

Digital Library

Google Scholar

[9]

R. Luk, H. Leong, T. Dillon, A. Chan, W. B. Croft, and J. Allan. A Survey in Indexing and Searching XML Documents. JASIS, 6(53):415--437, Mar. 2002.

Digital Library

Google Scholar

[10]

B. Piwowarski and M. Lalmas. Interface pour l'évaluation de systèmes de recherche sur des documents XML. In Premiere COnference en Recherche d'Information et Applications (CORIA'04), Toulouse, France, Mar. 2004. Hermès.

Google Scholar

[11]

K. Sparck~Jones and C. J. Van Rijsbergen. Report on the need for and provision of an ideal information retrieval test collection. Technical Report 5266, Computer Laboratory, University of Cambridge, Cambridge, England, 1975.

Google Scholar

[12]

E. M. Voorhees and D. K. Harman, editors. The Tenth Text Retrieval Conference (TREC 2001), Gaithersburg, MD, USA, 2002. NIST.

Crossref

Google Scholar

Cited By

View all

Arvola PVainio JJunkkari MKekäläinen JKamps JKraaij WFuhr N(2012)Model for simulating result document browsing in focused retrievalProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362764(238-241)Online publication date: 21-Aug-2012
https://dl.acm.org/doi/10.1145/2362724.2362764
Kazai GKamps JKoolen MMilic-Frayling NMa WNie JBaeza-Yates RChua TCroft W(2011)Crowdsourcing for book search evaluationProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval10.1145/2009916.2009947(205-214)Online publication date: 24-Jul-2011
https://dl.acm.org/doi/10.1145/2009916.2009947
Arvola PVainio J(2011)The Potential Benefit of Focused Retrieval in Relevant-in-Context TaskComparative Evaluation of Focused Retrieval10.1007/978-3-642-23577-1_2(33-43)Online publication date: 2011
https://doi.org/10.1007/978-3-642-23577-1_2
Show More Cited By

Index Terms

Providing consistent and exhaustive relevance assessments for XML retrieval evaluation

Recommendations

Sound and complete relevance assessment for XML retrieval

In information retrieval research, comparing retrieval approaches requires test collections consisting of documents, user requests and relevance assessments. Obtaining relevance assessments that are as sound and complete as possible is crucial for the ...
Hybrid XML Retrieval: Combining Information Retrieval and a Native XML Database
Abstract
This paper investigates the impact of three approaches to XML retrieval: using Zettair, a full-text information retrieval system; using eXist, a native XML database; and using a hybrid system that takes full article answers from Zettair and uses ...
Structural Relevance Feedback in XML Retrieval
FQAS '09: Proceedings of the 8th International Conference on Flexible Query Answering Systems

Contrarily to classical information retrieval systems, the systems that treat structured documents include the structural dimension through the document and query comparison. Thus, the retrieval of relevant results means the retrieval of document ...

Comments

Information & Contributors

Information

Published In

CIKM '04: Proceedings of the thirteenth ACM international conference on Information and knowledge management

November 2004

678 pages

ISBN:1581138741

DOI:10.1145/1031171

General Chair:
David Grossman
Illinois Institute of Technology
,
Program Chairs:
Luis Gravano
Columbia University
,
ChengXiang Zhai
University of Illinois at Urbana-Champaign
,
Otthein Herzog
University of Bremen, Germany
,
David A. Evans
Clairvoyance Corporation

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 13 November 2004

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

CIKM04

Sponsor:

CIKM04: Conference on Information and Knowledge Management

November 8 - 13, 2004

D.C., Washington, USA

Acceptance Rates

Overall Acceptance Rate 1,861 of 8,427 submissions, 22%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

17
Total Citations
View Citations
435
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Arvola PVainio JJunkkari MKekäläinen JKamps JKraaij WFuhr N(2012)Model for simulating result document browsing in focused retrievalProceedings of the 4th Information Interaction in Context Symposium10.1145/2362724.2362764(238-241)Online publication date: 21-Aug-2012
https://dl.acm.org/doi/10.1145/2362724.2362764
Kazai GKamps JKoolen MMilic-Frayling NMa WNie JBaeza-Yates RChua TCroft W(2011)Crowdsourcing for book search evaluationProceedings of the 34th international ACM SIGIR conference on Research and development in Information Retrieval10.1145/2009916.2009947(205-214)Online publication date: 24-Jul-2011
https://dl.acm.org/doi/10.1145/2009916.2009947
Arvola PVainio J(2011)The Potential Benefit of Focused Retrieval in Relevant-in-Context TaskComparative Evaluation of Focused Retrieval10.1007/978-3-642-23577-1_2(33-43)Online publication date: 2011
https://doi.org/10.1007/978-3-642-23577-1_2
Arvola PVainio J(2010)The potential benefit of focused retrieval in relevant-in-context taskProceedings of the 9th international conference on Initiative for the evaluation of XML retrieval: comparative evaluation of focused retrieval10.5555/2040369.2040372(33-43)Online publication date: 13-Dec-2010
https://dl.acm.org/doi/10.5555/2040369.2040372
Arvola PKekäläinen JJunkkari M(2010)Expected reading effort in focused retrieval evaluationInformation Retrieval10.1007/s10791-010-9133-913:5(460-484)Online publication date: 6-May-2010
https://doi.org/10.1007/s10791-010-9133-9
Lalmas M(2009)XML RetrievalSynthesis Lectures on Information Concepts, Retrieval, and Services10.2200/S00203ED1V01Y200907ICR0071:1(1-111)Online publication date: Jan-2009
https://doi.org/10.2200/S00203ED1V01Y200907ICR007
Piwowarski BTrotman ALalmas M(2008)Sound and complete relevance assessment for XML retrievalACM Transactions on Information Systems10.1145/1416950.141695127:1(1-37)Online publication date: 23-Dec-2008
https://dl.acm.org/doi/10.1145/1416950.1416951
Lalmas MTombros A(2007)INEX 2002-2006Proceedings of the 1st international conference on Digital libraries: research and development10.5555/1782334.1782357(187-196)Online publication date: 13-Feb-2007
https://dl.acm.org/doi/10.5555/1782334.1782357
Lalmas MTombros A(2007)Evaluating XML retrieval effectiveness at INEXACM SIGIR Forum10.1145/1273221.127322541:1(40-57)Online publication date: 1-Jun-2007
https://dl.acm.org/doi/10.1145/1273221.1273225
Ashoori ELalmas MTsikrika T(2007)Examining topic shifts in content-oriented XML retrievalInternational Journal on Digital Libraries10.1007/s00799-007-0026-58:1(39-60)Online publication date: 25-Oct-2007
https://dl.acm.org/doi/10.1007/s00799-007-0026-5
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

Sound and complete relevance assessment for XML retrieval

Hybrid XML Retrieval: Combining Information Retrieval and a Native XML Database

Structural Relevance Feedback in XML Retrieval

Comments

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Index Terms

Recommendations

Sound and complete relevance assessment for XML retrieval

Hybrid XML Retrieval: Combining Information Retrieval and a Native XML Database

Structural Relevance Feedback in XML Retrieval

Comments

Information

Published In

Sponsors

Publisher

Publication History

Permissions

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Get Access

Login options

Full Access

View options

PDF

eReader

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations