Extracting and applying evaluation criteria for ontology quality assessment

The purpose of this paper is to formulate apposite criteria for ontology evaluation and test them through assessments of existing ontologies.

A literature review provided the basis from which to extract the categories relevant to an evaluation of internal ontology components. According to the ontology evaluation categories, a panel of experts provided the evaluation criteria for each category via Delphi survey. Reliability was gauged by applying the criteria to assessments of existing smartphone ontologies.

Existing research tends to approach ontology evaluation through comparison with well-engineered ontologies, implementation in target applications and appropriateness/interconnection appraisals in relation to raw data, but such methodologies fall short of shedding light on the internal workings of ontologies, such as structure, semantic representation and interoperability. This study adopts its evaluation categories from previous research while also collecting concrete evaluation criteria from an expert panel and verifying the reliability of the resulting 53 criteria.

This is the first published study to extract ontology evaluation criteria in terms of syntax, semantics and pragmatics. The results can be used as an evaluation index following ontology construction.