article

Requirements-oriented methodology for evaluating ontologies

Authors:

Audrey TamAuthors Info & Claims

Information Systems, Volume 34, Issue 8

Pages 766 - 791

Published: 01 December 2009 Publication History

Abstract

Many applications benefit from the use of a suitable ontology but it can be difficult to determine which ontology is best suited to a particular application. Although ontology evaluation techniques are improving as more measures and methodologies are proposed, the literature contains few specific examples of cohesive evaluation activity that links ontologies, applications and their requirements, and measures and methodologies. In this paper, we present ROMEO, a requirements-oriented methodology for evaluating ontologies, and apply it to the task of evaluating the suitability of some general ontologies (variants of sub-domains of the Wikipedia category structure) for supporting browsing in Wikipedia. The ROMEO methodology identifies requirements that an ontology must satisfy, and maps these requirements to evaluation measures. We validate part of this mapping with a task-based evaluation method involving users, and report on our findings from this user study.

References

[1]

Baeza-Yates, R. and Ribeiro-Neto, B., Modern Information Retrieval. 1999. ACM Press, Addison-Wesley.

[2]

C. Brewster, H. Alani, S. Dasmahapatra, Y. Wilks, Data driven ontology evaluation, in: Proceedings of the 4th International Conference on Language Resources and Evaluation (LREC), Lisbon, Portugal, 2004. European Language Resources Association.

[3]

Davidson, E.J., Evaluation Methodology Basics. 2005. Sage Publications, London, UK.

[4]

deMarco, T., Controlling Software Projects: Management, Measurement, and Estimates. 1986. Prentice-Hall PTR, Upper Saddle River, NJ, USA.

[5]

Dong, J.S., Lee, C.H., Lee, H.B., Li, Y.F. and Wang, H., A combined approach to checking web ontologies. In: Proceedings of the International Conference on World Wide Web, ACM Press. pp. 714-722.

[6]

A. Gangemi, C. Catenacci, M. Ciaramita, J. Lehmann, Ontology evaluation and validation, Technical Report, Laboratory for Applied Ontology, 2005.

[7]

Gómez-Pérez, A., Towards a framework to verify knowledge sharing technology. Expert Systems With Applications. v11 i4. 519-529.

[8]

Gómez-Pérez, A., Evaluation of ontologies. International Journal of Intelligent Systems. v16. 391-409.

[9]

Gruber, T.R., A translation approach to portable ontology specifications. Knowledge Acquisition. v5 i2. 199-220.

[10]

Gruber, T.R., Toward principles for the design of ontologies used for knowledge sharing. International Journal Human-Computer Studies. v43 i5-6. 907-928.

[11]

M. Grüninger, M. Fox, Methodology for the design and evaluation of ontologies, in: Workshop on Basic Ontological Issues in Knowledge Sharing, IJCAI'95, 1995.

[12]

N. Guarino, Some ontological principles for designing upper level lexical resources, in: Proceedings of the 1st International Conference on Lexical Resources and Evaluation (LREC), May 1998.

[13]

Guarino, N., Towards a formal evaluation of ontology quality (in Why evaluate ontology technologies? Because they work!). IEEE Intelligent Systems. v19 i4. 74-81.

[14]

Guarino, N. and Welty, C., Evaluating ontological decisions with ontoclean. Communications of ACM. v45 i2. 61-65.

[15]

Haase, P., Hotho, A., Schmidt-Thieme, L. and Sure, Y., Collaborative and usage-driven evolution of personal ontologies. In: Proceedings of the 2nd European Semantic Web Conference, Lecture Notes in Computer Science, vol. 3532. Springer, London, UK. pp. 486-499.

[16]

Lethbridge, T., Metrics for concept-oriented knowledge bases. Software Engineering and Knowledge Engineering. v8 i2. 161-188.

[17]

Lozano-Tello, A. and Gómez-Pérez, A., OntoMetric: a method to choose the appropriate ontology. Journal of Database Management. v15 i2. 1-18.

[18]

Maedche, A. and Staab, S., Measuring similarity between ontologies. In: Proceedings of the 13th International Conference on Knowledge Engineering and Knowledge Management, Springer, London, UK. pp. 251-263.

[19]

Marchionini, G., Information Seeking in Electronic Environments. 1995. Cambridge University Press, Cambridge.

[20]

McGuinness, D.L., Spinning the semantic web: bringing the world wide web to its full potential. In: Ontologies Come of Age, MIT Press, Cambridge, MA. pp. 171-195.

[21]

Moeller, K. and Paulish, D., Software Metrics: A Practitioner's Guide to Improved Product Development. 1993. IEEE Press, New York.

[22]

Orme, A., Yao, H. and Etzkorn, L., Coupling metrics for ontology-based systems. IEEE Software. v23 i2. 102-108.

[23]

Saaty, T., How to make a decision: the analytic hierarchy process. Interfaces. v24 i6. 19-43.

[24]

Scriven, M., Evaluation Thesaurus. 1991. fourth ed. Sage Publications, Newbury Park, CA.

[25]

Sowa, J., Knowledge Representation: Logical Philosophical and Computational Foundations. 2000. Brooks Cole.

[26]

S. Tartir, I. Arpinar, M. Moore, A. Sheth, B. Aleman-Meza, OntoQA: metric-based ontology quality analysis, in: Proceedings of the Workshop on Knowledge Acquisition from Distributed, Autonomous, Semantically Heterogeneous Data and Knowledge Sources at IEEE International Conference on Data Mining (ICDM), Texas, USA, November 2005, pp. 45-53.

[27]

Thompson, R. and Croft, W., Support for browsing in an intelligent text retrieval system. International Journal of Man-Machine Studies. v30 i6. 639-668.

[28]

Welty, C. and Guarino, N., Supporting ontological analysis of taxonomic relationships. Data and Knowledge Engineering. v39 i1. 51-74.

[29]

Wikipedia online editing guidelines for categories {http://en.wikipedia.org/wiki/Wikipedia:Category}, Accessed January 2008.

[30]

Wikipedia guidelines for category proposals and implementations (original) {http://meta.wikimedia.org/wiki/Categorization_requirements}, Accessed January 2008.

[31]

J. Yu, Requirements-oriented methodology for evaluating ontologies, Ph.D. Thesis, RMIT University, Melbourne, Australia, 2008.

[32]

J. Yu, J. Thom, A. Tam, Ontology evaluation: using Wikipedia categories for browsing, in: Proceedings of the 16th Conference on Information and Knowledge Management, ACM Press, 2007, pp. 223-232.

[33]

J. Yu, J.A. Thom, A. Tam, Evaluating ontology criteria for requirements in a geographic travel domain, in: Proceedings of International Conference on Ontologies, DataBases and Applications of Semantics, 2005.

[34]

Zhao, Y. and Karypis, G., Hierarchical clustering algorithms for document datasets. Data Mining and Knowledge Discovery. v10 i2. 141-168.

Cited By

Yun WZhang XLi ZLiu HHan M(2021)Knowledge modelingInternational Journal of Intelligent Systems10.1002/int.2235736:4(1686-1720)Online publication date: 4-Jan-2021
https://dl.acm.org/doi/10.1002/int.22357
McDaniel MStorey V(2019)Evaluating Domain OntologiesACM Computing Surveys10.1145/332912452:4(1-44)Online publication date: 11-Sep-2019
https://dl.acm.org/doi/10.1145/3329124
Lara JGuerra E(2018)Refactoring Multi-Level ModelsACM Transactions on Software Engineering and Methodology10.1145/328098527:4(1-56)Online publication date: 16-Nov-2018
https://dl.acm.org/doi/10.1145/3280985
Show More Cited By

Recommendations

Ontology evaluation using wikipedia categories for browsing
CIKM '07: Proceedings of the sixteenth ACM conference on Conference on information and knowledge management

Ontology evaluation is a maturing discipline with methodologies and measures being developed and proposed. However, evaluation methods that have been proposed have not been applied to specific examples. In this paper, we present the state-of-the-art in ...
A semantic metrics suite for evaluating modular ontologies

Ontologies, which are formal representations of knowledge within a domain, can be used for designing and sharing conceptual models of enterprises information for the purpose of enhancing understanding, communication and interoperability. For ...
Some ideas and examples to evaluate ontologies
CAIA '95: Proceedings of the 11th Conference on Artificial Intelligence for Applications

Ontologies are the platforms that enable the sharing and reuse of knowledge by establishing common vocabularies and semantic interpretations of terms. While ontologies may provide for reusability, sharability or both, the evaluation of their definitions ...

Comments

Information & Contributors

Information

Published In

Copyright © Elsevier B.V. © 2009.

Publisher

Elsevier Science Ltd.

United Kingdom

Publication History

Published: 01 December 2009

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yun WZhang XLi ZLiu HHan M(2021)Knowledge modelingInternational Journal of Intelligent Systems10.1002/int.2235736:4(1686-1720)Online publication date: 4-Jan-2021
https://dl.acm.org/doi/10.1002/int.22357
McDaniel MStorey V(2019)Evaluating Domain OntologiesACM Computing Surveys10.1145/332912452:4(1-44)Online publication date: 11-Sep-2019
https://dl.acm.org/doi/10.1145/3329124
Lara JGuerra E(2018)Refactoring Multi-Level ModelsACM Transactions on Software Engineering and Methodology10.1145/328098527:4(1-56)Online publication date: 16-Nov-2018
https://dl.acm.org/doi/10.1145/3280985
Degbelo A(2017)A Snapshot of Ontology Evaluation Criteria and StrategiesProceedings of the 13th International Conference on Semantic Systems10.1145/3132218.3132219(1-8)Online publication date: 11-Sep-2017
https://dl.acm.org/doi/10.1145/3132218.3132219
Etudo UYoon VLiu D(2017)Financial Concept Element Mapper (FinCEM) for XBRL interoperabilityDecision Support Systems10.1016/j.dss.2017.04.00698:C(36-48)Online publication date: 1-Jun-2017
https://dl.acm.org/doi/10.1016/j.dss.2017.04.006
Li YThomas MOsei-Bryson K(2017)Ontology-based data mining model management for self-service knowledge discoveryInformation Systems Frontiers10.1007/s10796-016-9637-y19:4(925-943)Online publication date: 1-Aug-2017
https://dl.acm.org/doi/10.1007/s10796-016-9637-y
Fan SHua ZStorey VZhao J(2016)A process ontology based approach to easing semantic ambiguity in business process modelingData & Knowledge Engineering10.1016/j.datak.2016.01.001102:C(57-77)Online publication date: 1-Mar-2016
https://dl.acm.org/doi/10.1016/j.datak.2016.01.001
Narock TYoon VMarch S(2014)A provenance-based approach to semantic web service description and discoveryDecision Support Systems10.1016/j.dss.2014.04.00764:C(90-99)Online publication date: 1-Aug-2014
https://dl.acm.org/doi/10.1016/j.dss.2014.04.007
Rico MCaliusco MChiotti OGalli M(2014)OntoQualitasComputers in Industry10.1016/j.compind.2014.07.01065:9(1291-1300)Online publication date: 1-Dec-2014
https://dl.acm.org/doi/10.1016/j.compind.2014.07.010
Mortensen J(2013)Crowdsourcing Ontology VerificationProceedings of the 12th International Semantic Web Conference - Part II10.1007/978-3-642-41338-4_30(448-455)Online publication date: 21-Oct-2013
https://dl.acm.org/doi/10.1007/978-3-642-41338-4_30
Show More Cited By

View Options

View options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Issue’s Table of Contents