Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/2187980.2188223acmotherconferencesArticle/Chapter ViewAbstractPublication PagesthewebconfConference Proceedingsconference-collections
tutorial

Towards minimal test collections for evaluation of audio music similarity and retrieval

Published: 16 April 2012 Publication History

Abstract

Reliable evaluation of Information Retrieval systems requires large amounts of relevance judgments. Making these annotations is quite complex and tedious for many Music Information Retrieval tasks, so performing such evaluations requires too much effort. A low-cost alternative is the application of Minimal Test Collection algorithms, which offer quite reliable results while significantly reducing the annotation effort. The idea is to incrementally select what documents to judge so that we can compute estimates of the effectiveness differences between systems with a certain degree of confidence. In this paper we show a first approach towards its application to the evaluation of the Audio Music Similarity and Retrieval task, run by the annual MIREX evaluation campaign. An analysis with the MIREX 2011 data shows that the judging effort can be reduced to about 35% to obtain results with 95% confidence.

References

[1]
B. Carterette. Low-Cost and Robust Evaluation of Information Retrieval Systems. Ph.D. dissertation, Department of Computer Science, University of Massachusetts Amherst, 2008.
[2]
B. Carterette. Robust Test Collections for Retrieval Evaluation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 55--62, 2007.
[3]
B. Carterette, J. Allan, and R. Sitaraman. Minimal Test Collections for Retrieval Evaluation. In International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 268--275, 2006.
[4]
J.S. Downie. The Scientific Evaluation of Music Information Retrieval Systems: Foundations and Future. Computer Music Journal. 28(2): 12--23, 2004.
[5]
J.S. Downie, A.F. Ehmann, M. Bay, and M.C. Jones. . The Music Information Retrieval Evaluation eXchange: Some Observations and Insights. In Advances in Music Information Retrieval, W.R. Zbigniew and A.A. Wieczorkowska, eds. Springer. 2010, 93--115.
[6]
J. Urbano. Information Retrieval Meta-Evaluation: Challenges and Opportunities in the Music Domain. In International Society for Music Information Retrieval Conference, pages 609--614, 2011.
[7]
J. Urbano, D. Martín, M. Marrero, and J. Morato. Audio Music Similarity and Retrieval: Evaluation Power and Stability. In International Society for Music Information Retrieval Conference, pages 597--602, 2011.
[8]
E.M. Voorhees. Variations in Relevance Judgments and the Measurement of Retrieval Effectiveness. Information Processing and Management. 36(5): 697--716, 2000.
[9]
E.M. Voorhees and D.K. Harman. TREC: Experiment and Evaluation in Information Retrieval. MIT Press, 2005.

Cited By

View all
  • (2013)Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systemsInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0030-42:1(59-70)Online publication date: 1-Jan-2013

Index Terms

  1. Towards minimal test collections for evaluation of audio music similarity and retrieval

    Recommendations

    Comments

    Information & Contributors

    Information

    Published In

    cover image ACM Other conferences
    WWW '12 Companion: Proceedings of the 21st International Conference on World Wide Web
    April 2012
    1250 pages
    ISBN:9781450312301
    DOI:10.1145/2187980
    Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

    Sponsors

    • Univ. de Lyon: Universite de Lyon

    In-Cooperation

    Publisher

    Association for Computing Machinery

    New York, NY, United States

    Publication History

    Published: 16 April 2012

    Permissions

    Request permissions for this article.

    Check for updates

    Author Tags

    1. evaluation
    2. music information retrieval
    3. relevance judgments
    4. test collections

    Qualifiers

    • Tutorial

    Conference

    WWW 2012
    Sponsor:
    • Univ. de Lyon
    WWW 2012: 21st World Wide Web Conference 2012
    April 16 - 20, 2012
    Lyon, France

    Acceptance Rates

    Overall Acceptance Rate 1,899 of 8,196 submissions, 23%

    Contributors

    Other Metrics

    Bibliometrics & Citations

    Bibliometrics

    Article Metrics

    • Downloads (Last 12 months)1
    • Downloads (Last 6 weeks)0
    Reflects downloads up to 13 Jan 2025

    Other Metrics

    Citations

    Cited By

    View all
    • (2013)Minimal test collections for low-cost evaluation of Audio Music Similarity and Retrieval systemsInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0030-42:1(59-70)Online publication date: 1-Jan-2013

    View Options

    Login options

    View options

    PDF

    View or Download as a PDF file.

    PDF

    eReader

    View online with eReader.

    eReader

    Media

    Figures

    Other

    Tables

    Share

    Share

    Share this Publication link

    Share on social media