research-article

Evaluation with the VIRTUOSO platform: an open source platform for information extraction and retrieval evaluation

Authors:

Gérard M. Dupont,

Gaël de Chalendar,

Khaled Khelif,

Dmitri Voitsekhovitch,

Géraud Canet,

Stéphan BrunessauxAuthors Info & Claims

DESIRE '11: Proceedings of the 2011 workshop on Data infrastructurEs for supporting information retrieval evaluation

Pages 13 - 18

https://doi.org/10.1145/2064227.2064254

Published: 28 October 2011 Publication History

Get Access

Abstract

This paper describes a software architecture designed to enable the evaluation of information processing and retrieval systems. The overall objective of our project is to provide an open technical framework for the integration of tools for collection, processing, analysis and communication of open source information¹. However, enabling the integration of heterogeneous components does not make sense without a proper way to compare the capabilities of multiple tools.

Thus, as part of the architecture the VIRTUOSO platform offers an evaluation framework which allows one to deploy and run evaluation kits for different use-cases.

References

[1]

G. Canet, G. de Chalendar, L. Dubost, G. Dupont, A. Dyevre, and K. Khelif. Toward a versatile information toolkit for end-users oriented open-sources exploitation : VIRTUOSO. In "Sources Ouvertes et Services" - SOS'2011, En association avec EGC'2011, 2011.

Google Scholar

[2]

C. W. Cleverdon. Report on the testing and analysis of an investigation into the comparative efficiency of indexing systems. Technical report, Cranfield Coll. of Aeronautics, Cranfield, England, 1962.

Google Scholar

[3]

M. Dittenbach, B. Pflugfelder, A. Pesenhofer, G. Roda, and H. Berger. Soire: a service-oriented ir evaluation architecture. In Proceeding of the 18th ACM conference on Information and knowledge management, CIKM '09, pages 2101--2102, New York, NY, USA, 2009. ACM.

Digital Library

Google Scholar

[4]

S. T. Dumais and N. Belkin. TREC: Experiment and Evaluation in Information Retrieval, chapter The Interactive TREC Track: Putting the user into search., pages 123--153. E. Voorhees and D. Harman (Eds.), MIT Press, 2005.

Google Scholar

[5]

P. Giroux, S. Brunessaux, S. Brunessaux, J. Doucy, G. Dupont, B. Grilheres, Y. Mombrun, and A. Saval. Weblab: An integration infrastructure to ease the development of multimedia processing applications. ICSSEA, 2008.

Google Scholar

[6]

R. W. Group. Resource description framework (rdf): Concepts and abstract syntax. Technical report, W3C, 2004.

Google Scholar

[7]

P. Thompson, H. Turtle, B. Yang, and J. Flood. Trec-3 ad-hoc retrieval and routing experiments using the win system. In Gaithersburg, editor, Proc. of the Text REtrieval Conference, pages 211--218, USA, 1995.

Google Scholar

Cited By

View all

Ghioni RTaddeo MFloridi L(2023)Open source intelligence and AI: a systematic review of the GELSI literatureAI & SOCIETY10.1007/s00146-023-01628-x39:4(1827-1842)Online publication date: 28-Jan-2023
https://doi.org/10.1007/s00146-023-01628-x
Mordinyi RSerral EEkaputra F(2016)Semantic Data Integration: Tools and ArchitecturesSemantic Web Technologies for Intelligent Engineering Applications10.1007/978-3-319-41490-4_8(181-217)Online publication date: 15-Nov-2016
https://doi.org/10.1007/978-3-319-41490-4_8
Mordinyi RSerral EWinkler DBiffl S(2014)Evaluating software architectures using ontologies for storing and versioning of engineering data in heterogeneous systems engineering environmentsProceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA)10.1109/ETFA.2014.7005237(1-10)Online publication date: Sep-2014
https://doi.org/10.1109/ETFA.2014.7005237
Show More Cited By

Index Terms

Recommendations

A functional component framework for integration infrastructures

Enterprises cooperate extensively with other enterprises in various forms. These collaboration forms need suitable infrastructures and supporting applications to realise the collaboration. This paper presents ideas and requirements for an integration ...
Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection
SP '11: Proceedings of the 2011 IEEE Symposium on Security and Privacy

Introspection has featured prominently in many recent security solutions, such as virtual machine-based intrusion detection, forensic memory analysis, and low-artifact malware analysis. Widespread adoption of these approaches, however, has been hampered ...
An Extendable OpenSource Architecture of E-Learning System
ECONF '13: Proceedings of the 2013 Fourth International Conference on e-Learning "Best Practices in Management, Design and Development of e-Courses: Standards of Excellence and Creativity"

In this paper, we present an extendable open source architecture and model of an e-Learning system. The system composed of essential e-Learning core components and optional open source extensions. Open source extensions, developed and maintain by open ...

Comments

Information & Contributors

Information

Published In

DESIRE '11: Proceedings of the 2011 workshop on Data infrastructurEs for supporting information retrieval evaluation

October 2011

44 pages

ISBN:9781450309523

DOI:10.1145/2064227

Program Chairs:
Maristella Agosti
University of Padua, Italy
,
Nicola Ferro
University of Padua, Italy
,
Costantino Thanos
ISTI-CNR, Pisa, Italy

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2011

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

CIKM '11

Sponsor:

CIKM '11: International Conference on Information and Knowledge Management

October 28, 2011

Glasgow, Scotland, UK

Upcoming Conference

CIKM '25

Sponsor:
sigir
sigir

The 34th ACM International Conference on Information and Knowledge Management

November 10 - 14, 2025

Seoul , Republic of Korea

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
146
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Ghioni RTaddeo MFloridi L(2023)Open source intelligence and AI: a systematic review of the GELSI literatureAI & SOCIETY10.1007/s00146-023-01628-x39:4(1827-1842)Online publication date: 28-Jan-2023
https://doi.org/10.1007/s00146-023-01628-x
Mordinyi RSerral EEkaputra F(2016)Semantic Data Integration: Tools and ArchitecturesSemantic Web Technologies for Intelligent Engineering Applications10.1007/978-3-319-41490-4_8(181-217)Online publication date: 15-Nov-2016
https://doi.org/10.1007/978-3-319-41490-4_8
Mordinyi RSerral EWinkler DBiffl S(2014)Evaluating software architectures using ontologies for storing and versioning of engineering data in heterogeneous systems engineering environmentsProceedings of the 2014 IEEE Emerging Technology and Factory Automation (ETFA)10.1109/ETFA.2014.7005237(1-10)Online publication date: Sep-2014
https://doi.org/10.1109/ETFA.2014.7005237
Agosti MFerro NThanos C(2012)DESIRE 2011ACM SIGIR Forum10.1145/2215676.221568146:1(51-55)Online publication date: 20-May-2012
https://dl.acm.org/doi/10.1145/2215676.2215681

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Cited By

Index Terms

Recommendations

A functional component framework for integration infrastructures

Virtuoso: Narrowing the Semantic Gap in Virtual Machine Introspection

An Extendable OpenSource Architecture of E-Learning System