Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.1145/2064227.2064257acmconferencesArticle/Chapter ViewAbstractPublication PagescikmConference Proceedingsconference-collections
research-article

Infrastructure and workflow for the formal evaluation of semantic search technologies

Published: 28 October 2011 Publication History

Abstract

This paper describes an infrastructure for the automated evaluation of semantic technologies and, in particular, semantic search technologies. For this purpose, we present an evaluation framework which follows a service-oriented approach for evaluating semantic technologies and uses the Business Process Execution Language (BPEL) to define evaluation workflows that can be executed by process engines. This framework supports a variety of evaluations, from different semantic areas, including search, and is extendible to new evaluations. We show how BPEL addresses this diversity as well as how it is used to solve specific challenges such as heterogeneity, error handling and reuse.

References

[1]
A. Alves, A. Arkin, S. Askary, B. Bloch, F. Curbera, Y. Goland, N. Kartha, Sterling, D. König, V. Mehta, S. Thatte, D. van der Rijn, P. Yendluri, and A. Yiu. Web services business process execution language version 2.0. OASIS Standard Committee, April 2007.
[2]
M. Esteban-Gutiérrez, R. García-Castro, and A. Gómez-Pérez. Executing evaluations over semantic technologies using the SEALS Platform. In Proceedings of the International Workshop on Evaluation of Semantic Technologies (IWEST 2010).
[3]
J. Euzenat et al. Results of the ontology alignment evaluation initiative 2010. In Proc. of the 5th Workshop on Ontology Matching (OM-2010), collocated with ISWC-2010, pages 85--117, Shanghai (CN), 2010.
[4]
R. García-Castro, M. Esteban-Gutiérrez, and A. Gómez-Pérez. Towards an infrastructure for the evaluation of semantic technologies. In Proceedings of the eChallenges 2010 Conference, Warsaw, Poland, October 27--29 2010.
[5]
R. García-Castro and A. Gómez-Pérez. RDF(S) interoperability results for semantic web technologies. International Journal of Software Engineering and Knowledge Engineering, 19(8):1083--1108, December 2009.
[6]
R. García-Castro and A. Gómez-Pérez. Interoperability results for Semantic Web technologies using OWL as the interchange language. Web Semantics: Science, Services and Agents on the World Wide Web, 8:278--291, November 2010.
[7]
I. Horrocks and P. F. Patel-Schneider. DL systems comparison. In Proc. of the 1998 Description Logic Workshop (DL'98), volume 11, pages 55--57, 1998.
[8]
IEEE. IEEE 1061-1998. IEEE Standard for a Software Quality Metrics Methodology. IEEE, December 1998.
[9]
E. Kaufmann. Talking to the Semantic Web - Natural Language Query Interfaces for Casual End-Users. PhD thesis, Faculty of Economics, Business Administration and Information Technology of the University of Zurich, September 2007.
[10]
M. Klusch, A. Leger, D. Martin, M. Paolucci, A. Bernstein, and U. Kuester. Annual International Contest S3 on Semantic Service Selection. http://www-ags.dfki.uni-sb.de/~klusch/s3/.
[11]
F. Massacci and F. M. Donini. Design and results of TANCS-2000 non-classical (modal) systems comparison. In TABLEAUX '00: Proceedings of the International Conference on Automated Reasoning with Analytic Tableaux and Related Methods, pages 52--56, London, UK, 2000. Springer-Verlag.
[12]
L. Nixon, R. García-Castro, S. N. Wrigley, M. Yatskevich, C. T. D. Santos, and L. Cabral. The state of semantic technology today - overview of the first SEALS evaluation campaigns. In I-SEMANTICS 2011 - 7th International Conference on Semantic Systems, 2011.
[13]
C. Petrie, T. Margaria, H. Lausen, and M. Zaremba. Semantic Web Services Challenge: Results from the First Year. Springer, 2009.
[14]
V. Uren, Y. Lei, V. Lopez, H. Liu, E. Motta, and M. Giordanino. The usability of semantic search tools: a review. The Knowledge Engineering Review, 22(4):361--377, 2007.
[15]
S. N. Wrigley, K. Elbedweihy, D. Reinhard, A. Bernstein, and F. Ciravegna. Evaluating semantic search tools using the SEALS Platform. In Proceedings of the International Workshop on Evaluation of Semantic Technologies (IWEST 2010) Workshop, collocated at ISWC 2010, 2010.
[16]
S. N. Wrigley, D. Reinhard, K. Elbedweihy, A. Bernstein, and F. Ciravegna. Methodology and campaign design for the evaluation of semantic search tools. In Proceedings of the International Workshop on Semantic Search, collocated at WWW 2010, 2010.

Cited By

View all

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
DESIRE '11: Proceedings of the 2011 workshop on Data infrastructurEs for supporting information retrieval evaluation
October 2011
44 pages
ISBN:9781450309523
DOI:10.1145/2064227
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 28 October 2011

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. bpel
  2. evaluation infrastructure
  3. semantic search
  4. semantic technology evaluation
  5. workflows

Qualifiers

  • Research-article

Conference

CIKM '11
Sponsor:

Upcoming Conference

CIKM '25

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)0
  • Downloads (Last 6 weeks)0
Reflects downloads up to 04 Jan 2025

Other Metrics

Citations

Cited By

View all

View Options

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media