Abstract
In this paper we compare the effectiveness scores and system rankings obtained with the inex-2002 metric, the official measure of INEX 2004, and the XCG metrics proposed in [4] and further developed here. For the comparisons, we use simulated runs as we can easily derive the desired system rankings that a reliable measure should produce based on a predefined set of user preferences. The results indicate that the XCG metrics are better suited for comparing systems for the INEX content-only (CO) task, where systems aim to return the highest scoring elements according to the user preferences reflected in a quantisation function, while also aiming to avoid returning overlapping components.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Cooper, W.: Expected search length: A single measure of retrieval effectiveness based on the weak ordering action of retrieval systems. American Documentation 19(1), 30–41 (1968)
Gövert, N., Kazai, G.: Overview of the INitiative for the Evaluation of XML Retrieval (INEX) 2002. In: Fuhr, N., Gövert, N., Kazai, G., Lalmas, M. (eds.) Proceedings of the First Workshop of the INitiative for the Evaluation of XML Retrieval (INEX), Dagstuhl, Germany, December 8–11 (2002); In: ERCIM Workshop Proceedings, Sophia Antipolis, France. ERCIM, pp. 1–17 (March 2003), http://www.ercim.org/publication/ws-proceedings/INEX2002.pdf
Järvelin, K., Kekäläinen, J.: Cumulated Gain-based evaluation of IR techniques. ACM Transactions on Information Systems (ACM TOIS) 20(4), 422–446 (2002)
Kazai, G.: Report of the inex 2003 metrics working group. In: Fuhr, N., Lalmas, M., Malik, S. (eds.) Proceedings of the 2nd Workshop of the INitiative for the Evaluation of XML retrieval (INEX), Dagstuhl, Germany, December 2003, pp. 184–190 (April 2004)
Kazai, G., Lalmas, M., de Vries, A.: The overlap problem in content-oriented XML retrieval evaluation. In: Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, Sheffield, UK, July 2004, pp. 72–79. ACM, New York (2004)
Kazai, G., Masood, S., Lalmas, M.: A study of the assessment of relevance for the inex 2002 test collection. In: McDonald, S., Tait, J.I. (eds.) ECIR 2004. LNCS, vol. 2997, pp. 296–310. Springer, Heidelberg (2004)
Kekäläinen, J., Järvelin, K.: Using graded relevance assessments in IR evaluation. Journal of the American Society for Information Science and Technology 53(13), 1120–1129 (2002)
Raghavan, V., Bollmann, P., Jung, G.: A critical investigation of recall and precision. ACM Transactions on Information Systems 7(3), 205–229 (1989)
Tombros, T., Larsen, B., Malik, S.: The interactive track at INEX 2004. In: Fuhr, N., Lalmas, M., Malik, S., Szlavik, Z. (eds.) Proceedings of the 3rd Workshop of the INitiative for the Evaluation of XML retrieval (INEX), Dagstuhl, Germany, December 2004 (2005)
Voorhees, E.M.: Overview of the TREC 2003 question answering track. In: Text REtrieval Conference, Gaithersburg (2003)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Kazai, G., Lalmas, M., de Vries, A. (2005). Reliability Tests for the XCG and inex-2002 Metrics. In: Fuhr, N., Lalmas, M., Malik, S., Szlávik, Z. (eds) Advances in XML Information Retrieval. INEX 2004. Lecture Notes in Computer Science, vol 3493. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11424550_5
Download citation
DOI: https://doi.org/10.1007/11424550_5
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-26166-7
Online ISBN: 978-3-540-32053-1
eBook Packages: Computer ScienceComputer Science (R0)