Article

Benchmarking image and video retrieval: an overview

Authors:

Stéphane Marchand-Maillet,

Marcel WorringAuthors Info & Claims

MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval

Pages 297 - 300

https://doi.org/10.1145/1178677.1178718

Published: 26 October 2006 Publication History

Abstract

Multimedia Information Retrieval (IR)techniques and associated systems are now numerous and justify the development of strategies and actions to objectively evaluate their capabilities. A number of initiatives following this line exist, each with its own peculiarities. In this paper, we take a bird's eye view on benchmarking multimedia IR systems (with the particular case of image and video retrieval)and summarize contributions made to a dedicated special session at the ACM Multimedia Information Retrieval Workshop (ACM MIR2006).From the analysis of a classical IR system, we identify locations of interest for evaluation of performance.We review proposals made in the context of existing benchmarks, each specialized in its own aspect and media.

References

[1]

Ricardo Baeza-Yates and Berthier Ribeiro-Neto, editors. Modern Information Retrieval Addison Wesley Longman Publishing Co. Inc., 1999.

Digital Library

[2]

P. Borlund. The IIR evaluation model: a framework for evaluation of interactive information retrieval systems. Information Research 8(3), 2003.

[3]

C. Buckley and E. M. Voorhees. Retrieval evaluation with incomplete information. In Proceedings of SIGIR 2004 New York, NY, USA, 2004.

Digital Library

[4]

C. W. Cleverdon. The Cranfield tests ion index language devices. ASLib proceedings 19(6):173--192, 1967.

[5]

C. Fluhr, P-AMoëllic, and P. Hede. Usage-oriented multimedia information retrieval technological evaluation. In Proceeding of the ACM Workshop on Multimedia Information Retrieval (MIR2006), special session on "Benchmarking Image and Video Retrieval Systems", Santa Barbara, CA, 2006.

Digital Library

[6]

Cormack G. V. and Lynam T. R. Statistical precision of information retrieval evaluation. In SIGIR 2006 Seattle, USA, 2006.

Digital Library

[7]

The International Music Information Retrieval Systems Evaluation Laboratory (IMIRSEL)Project. http://www.music-ir.org/evaluation

[8]

G. Marchionini. Human performance measures for video retrieval. In Proceeding of the ACM Workshop on Multimedia Information Retrieval (MIR2006), special session on "Benchmarking Image and Video Retrieval Systems", Santa Barbara, CA, 2006.

Digital Library

[9]

ImageCLEF: the image retrieval track within CLEF. http://ir.shef.ac.uk/imageclef/.

[10]

MIREX: Music Information Retrieval Evaluation eXchange. http://www.music-ir.org/mirex2006

[11]

Henning Müller, Stéphane Marchand-Maillet, and Thierry Pun. The truth about Corel -- Evaluation in image retrieval. In Proceedings of CIVR2002 London, UK, July 2002.

Digital Library

[12]

Henning Müller, Wolfgang Müller, David McG. Squire, Stéphane Marchand-Maillet, and Thierry Pun. Performance evaluation in content-based image retrieval:Overview and proposals. Pattern Recognition Letters (Special Issue on Image and Video Indexing) 22(5):593--601, 2001. H. Bunke and X. Jiang Eds.

Digital Library

[13]

G. P. Nguyen and M. Worring. Scenario optimization for interactive category search.In Proceedings of the 7th ACM SIGMM International Workshop on Multimedia Information Retrieval Singapore, 2005.

Digital Library

[14]

A. F. Smeaton, P. Over, and W. Kraaij. Evaluation campaigns and TRECVid.In Proceeding of the ACM Workshop on Multimedia Information Retrieval (MIR2006), special session on "Benchmarking Image and Video Retrieval Systems", Santa Barbara, CA, 2006.

Digital Library

[15]

Ellen M. Voorhees and Donna K. Harman, editors. TREC: Experiment and Evaluation in Information Retrieval Digital Libraries and Electronic Publishing. MIT Press, 2005.

Digital Library

[16]

T. Westerveld and R. W. Zwol. Benchmarking multimedia search in structured collections. In Proceeding of the ACM Workshop on Mul timedia Information Retrieval (MIR2006), special session on "Benchmarking Image and Video Retrieval Systems", Santa Barbara, CA, 2006.

Digital Library

[17]

Ryen W. White, Joemon M. Jose,and Ian Ruthven. Adapting to evolving needs:Evaluating a behaviour-based search interface. In 17th Annual Human-Computer Interaction Conference (HCI 2003) New Orleans, U.S.A, 2003.

Cited By

Rabl TFrank MDanisch MJacobsen HGowda B(2015)The Vision of BigBench 2.0Proceedings of the Fourth Workshop on Data analytics in the Cloud10.1145/2799562.2799642(1-4)Online publication date: 31-May-2015
https://dl.acm.org/doi/10.1145/2799562.2799642
Pegus PCecchet EShenoy POoi WFeng WLiu F(2015)Video BenchLabProceedings of the 6th ACM Multimedia Systems Conference10.1145/2713168.2723145(165-176)Online publication date: 18-Mar-2015
https://dl.acm.org/doi/10.1145/2713168.2723145
Thomee BLew M(2012)Interactive search in image retrieval: a surveyInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0014-41:2(71-86)Online publication date: 8-Jun-2012
https://doi.org/10.1007/s13735-012-0014-4
Show More Cited By

Index Terms

Benchmarking image and video retrieval: an overview
1. Information systems
  1. Information retrieval

Recommendations

Multimedia Retrieval Conference Enjoys Texas Hospitality

The Third ACM International Conference on Multimedia Retrieval (ICMR) was held in Dallas, Texas, from 16-19 April 2013. The conference aims to promote intellectual exchanges and interactions among scientists, engineers, students, multimedia researchers ...
Content-based image and video indexing and retrieval
Proceedings of the 2005 joint Chinese-German conference on Cognitive systems

This paper surveys some of the existing techniques and systems for content-based indexing and retrieval of two main types of multimedia data - images and videos. Furthermore, some of our recent work are reported. Specifically, in content-based image ...
An Image Clustering and Feedback-based Retrieval Framework

Most existing object-based image retrieval systems are based on single object matching, with its main limitation being that one individual image region object can hardly represent the user's retrieval target, especially when more than one object of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MIR '06: Proceedings of the 8th ACM international workshop on Multimedia information retrieval

October 2006

344 pages

ISBN:1595934952

DOI:10.1145/1178677

General Chairs:
James Z. Wang
The Pennsylvania State University
,
Nozha Boujemaa
INRIA Rocquencourt, France
,
Yixin Chen
The University of Mississippi

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 26 October 2006

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Conference

MM06

Sponsor:

MM06: The 14th ACM International Conference on Multimedia 2006

October 26 - 27, 2006

California, Santa Barbara, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
450
Total Downloads

Downloads (Last 12 months)8
Downloads (Last 6 weeks)1

Reflects downloads up to 26 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rabl TFrank MDanisch MJacobsen HGowda B(2015)The Vision of BigBench 2.0Proceedings of the Fourth Workshop on Data analytics in the Cloud10.1145/2799562.2799642(1-4)Online publication date: 31-May-2015
https://dl.acm.org/doi/10.1145/2799562.2799642
Pegus PCecchet EShenoy POoi WFeng WLiu F(2015)Video BenchLabProceedings of the 6th ACM Multimedia Systems Conference10.1145/2713168.2723145(165-176)Online publication date: 18-Mar-2015
https://dl.acm.org/doi/10.1145/2713168.2723145
Thomee BLew M(2012)Interactive search in image retrieval: a surveyInternational Journal of Multimedia Information Retrieval10.1007/s13735-012-0014-41:2(71-86)Online publication date: 8-Jun-2012
https://doi.org/10.1007/s13735-012-0014-4
Perkiö JTuominen AVähäkangas TMyllymäki P(2012)Image similarityMultimedia Tools and Applications10.1007/s11042-010-0562-757:1(5-27)Online publication date: 1-Mar-2012
https://dl.acm.org/doi/10.1007/s11042-010-0562-7
Budikova PBatko MZezula P(2011)Evaluation platform for content-based image retrieval systemsProceedings of the 15th international conference on Theory and practice of digital libraries: research and advanced technology for digital libraries10.5555/2042536.2042556(130-142)Online publication date: 26-Sep-2011
https://dl.acm.org/doi/10.5555/2042536.2042556
Sandhaus PBoll S(2011)Semantic analysis and retrieval in personal and social photo collectionsMultimedia Tools and Applications10.1007/s11042-010-0673-151:1(5-33)Online publication date: 1-Jan-2011
https://dl.acm.org/doi/10.1007/s11042-010-0673-1
Budikova PBatko MZezula P(2011)Evaluation Platform for Content-Based Image Retrieval SystemsResearch and Advanced Technology for Digital Libraries10.1007/978-3-642-24469-8_15(130-142)Online publication date: 2011
https://doi.org/10.1007/978-3-642-24469-8_15
Beecks CUysal MSeidl TLi SGao XSebe N(2010)Signature Quadratic Form DistanceProceedings of the ACM International Conference on Image and Video Retrieval10.1145/1816041.1816105(438-445)Online publication date: 5-Jul-2010
https://dl.acm.org/doi/10.1145/1816041.1816105
Beecks CUysal MSeidl T(2010)A comparative study of similarity measures for content-based multimedia retrieval2010 IEEE International Conference on Multimedia and Expo10.1109/ICME.2010.5582949(1552-1557)Online publication date: Jul-2010
https://doi.org/10.1109/ICME.2010.5582949
Wang LLeung C(2010)A model for predicting image search engine behaviourThe 3rd International Conference on Information Sciences and Interaction Sciences10.1109/ICICIS.2010.5534815(306-311)Online publication date: Jun-2010
https://doi.org/10.1109/ICICIS.2010.5534815
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents