Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
https://doi.org/10.1186/s12911-019-0859-z
R ES EA R CH
Open Access
Architecture and usability of
OntoKeeper, an ontology evaluation tool
Muhammad Amith1 , Frank Manion2 , Chen Liang3 , Marcelline Harris2 , Dennis Wang4 , Yongqun He5 and
Cui Tao1*
From The Third International Workshop on Semantics-Powered Data Analytics (SEPDA 2018)
Madrid, Spain. 03 December 2018
Abstract
Background: The existing community-wide bodies of biomedical ontologies are known to contain quality and
content problems. Past research has revealed various errors related to their semantics and logical structure.
Automated tools may help to ease the ontology construction, maintenance, assessment and quality assurance
processes. However, there are relatively few tools that exist that can provide this support to knowledge engineers.
Method: We introduce OntoKeeper as a web-based tool that can automate quality scoring for ontology developers.
We enlisted 5 experienced ontologists to test the tool and then administered the System Usability Scale to measure
their assessment.
Results: In this paper, we present usability results from 5 ontologists revealing high system usability of OntoKeeper,
and use-cases that demonstrate its capabilities in previous published biomedical ontology research.
Conclusion: To the best of our knowledge, OntoKeeper is the first of a few ontology evaluation tools that can help
provide ontology evaluation functionality for knowledge engineers with good usability.
Keywords: Biomedical ontologies, Ontology auditing, Quality evaluation, Usability analysis, Knowledge
management, Knowledge engineering, Semiotics, Semantic web
Background
Ontology evaluation is an important process in the development and maintenance for ontological knowledgebases, a process that helps ontologists determine if the
ontology is of good quality. Literature suggests that
readily-accessible and easily usable tooling are needed to
assist ontology developers with ontology evaluation task.
In this paper, we discuss the design of OntoKeeper, a
semiotic-driven ontology evaluation tool. We also discuss results of a usability evaluation by fellow ontologists and their overall assessment of the tool. Furthermore, OntoKeeper’s functionality has been demonstrated
in published studies as a tailorable and straightforward
method to validate biomedical ontologies. We conclude
*Correspondence: cui.tao@uth.tmc.edu
School of Biomedical Informatics, The University of Texas Health Science
Center at Houston, 7000 Fannin Street, Suite 600, 77030 Houston, TX, USA
Full list of author information is available at the end of the article
1
with future directions that will further improve on this
software service.
Ontologies have grown considerably over the last
decade. From observation, the Linked Open Data Cloud
[1] shows that most of the ontologies online are in the
life sciences. However, with all the ontological knowledge
bases, there are some considerations — such as maintenance (in terms of updating and upkeep) and quality.
Ontological quality entails a variety of issues to verify and
validate — logical consistency, veracity of the knowledge,
domain coverage, etc.
Based on a sample study of National Center for Biomedical Ontologies (NCBO) BioPortal ontologies, we have
shown that most ontologies do not demonstrate documented evidence of evaluation at the time of development
[2]. A seminal paper on ontology evaluation by Brank [3]
notes that the future of ontology evaluation and quality assessment will lie in automated tools to assist in the
© The Author(s). 2019 Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0
International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and
reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the
Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver
(http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
process. To the best our knowledge, there are no tools in
active use, nor are there standardized methods to evaluate or audit ontologies. We have positioned OntoKeeper,
our evaluation software, to facilitate quality evaluation of
ontologies and address the gap in evaluation and quality
control tools for ontologies.
Existing tools and research
We have previously noted that many NCBO ontologies
do not display documented evidence of evaluation, perhaps indicating lack of validating and verification of the
underlying ontological knowledge [4]. This suggests that
software tools for ontology evaluation are not readily
assessable by the ontology research community, or when
available, are not easy to use. To determine what tools
and methods are available to the community we queried
existing research databases the Association of Computing Machinery (ACM) and Institute of Electrical and
Electronics Engineers (IEEE) using “ontology evaluation”
or “ontology metrics” for papers published since 2007.
The query retrieved 92 unique papers from ACM Digital
Library and IEEE Xplore Digital Library. We reviewed the
abstract and the body of each paper based on an inclusion
criteria of relevancy for automated or semi-automated
ontology evaluation software tools. Worth noting, 35
papers discussed methodologies, experimental methods
or new metrics to evaluate ontologies. The results of
our review after our inclusion criteria yielded six papers
that introduced automated or semi-automated tools for
ontology evaluation (Table 1).
Table 1 Papers surveyed for ontology evaluation software tools
Paper
Method
Ontology Evaluation and Ranking
using OntoQA [5]
OntoQA metrics [6]
A Web-Based Ontology Evaluation
System [7]
Burton-Jones based; focused on
the “subjective” metrics
A Survey on Ontology Evaluation
Tools [8]
Survey paper that discussed
OntoAnalyser (OntoEdit plugin),
OntoGenerator (OntoEdit plugin),
WebODE plugin for OntoClean,
Ontology Evaluation Tool, and
S-OntoEval
Quality Model and Metrics of
Ontology for Semantic
Descriptions of Web Services [9]
Paper is corrected version [10] that
extends the ontology evaluation
framework they introduced earlier
An Ontology Selection and Ranking Applies analytic hierarchy process
System Based on the Analytic
to evaluate ontology through
Hierarchy Process [11]
Java-based application tools.
Calculates language expressivity,
domain coverage, size, consistency,
and cohesion
Ranking ontologies in the Ontology Ranking-based metric system
Building Competition BOC 2014
implemented as a web-based tool.
[12]
Calculates structural, semantic, and
term quality
Page 2 of 18
Ontology evaluation and ranking using OntoQA [5]
Tartir and Arpinar introduced OntoQA, which is a metric suite developed for ranking ontologies. The metric
suite was implemented through a Java-based web application. The metrics evaluates the ontology on two levels
- a “schema” level for structural intrinsic aspect and the
instance level for the data from the ontology’s knowledge base. In combination with the metrics and integrated
search results from Swoogle, the tool ranks the ontology
with respect to the results from Swoogle.
A web-based ontology evaluation system [7] Jianliang
and Xiaowei offered a server-side application that evaluates an ontology using crowdsourcing, with the intent
of providing a subjective evaluation of ontologies. Their
application tool utilized Burton-Jones and colleagues’
metric suite [13] to provide some subjective measurement
of the ontology. In accompaniment with visualization
of a concept from the ontology, the crowdsourced user
assessed the ontology based on a Likert scale for some of
the individual semiotic metrics.
A survey on ontology evaluation tools [8] A Survey on
Ontology Evaluation Tool by Aruna, et al., was a student
paper reviewing a selection of ontology evaluation software tools in conjunction with a set of properties serving
as an evaluation criteria - nine properties (two ontologyrelated and seven related to software performance). The
authors evaluated OntoAnalyser, OntoGenerator, OntoClean in WebODE, ONE-T, and S-OntoEval. Four of
the tools met their framework criteria for ontologyrelated properties - assessing syntactic quality (“Language
Conformity”) and semantic quality (“Consistency”). Only
OntoGenerator was cited in having better software performance.
Quality model and metrics of ontology for semantic
descriptions of web services [9] Zhu, et al., discussed
a web service called ASWebService that supports their
own set of ontology metrics grouped into a set of aspects
- Content, Presentation, Usage. The evaluation utilized
the metrics to compare an ontology with a gold standard
ontology to measure each quality attribute from their metrics. The authors’ future direction is to fully automate the
evaluation process.
An ontology selection and ranking system based on
the analytic hierarchy process [11] Groza, et al., introduced a Java desktop application that incorporates Analytical Hierarchical Process (AHP) framework to evaluate
and rank ontologies. AHP is a decision hierarchical tree
by Thomas Sayat that uses a set of criteria included produce numerical values for specific options [14]. In the
context of ontology evaluation, the criteria is a “language
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 3 of 18
expressivity”, “domain coverage”, “size”, “consistency”, and
“cohesion”. The resulting numerical values are assigned
to a set of examined ontologies. The tool reports the
evaluation and domain coverage results.
evaluation. In the following section, we recall our previous works where we used semiotic metrics [13] to evaluate
biomedical ontologies and how these use-cases informed
the design of our own software tool, OntoKeeper.
Ranking ontologies in the ontology building competition BOC 2014 [12] Jimborean and Gorza discussed
their web-based ontology evaluation and ranking system.
The tool incorporated a metric suite that was inspired
from OntoQA and various ontology ranking approaches
like AktiveRank [15] and OS_Rank [16]. The web-based
tool was developed in Java and utilizes OWL-API [17]
to extract data from the ontology and Apache Jena [18]
for SPARQL querying. The latter was used to support
competency questions. Users can attain a ranking score
and metric score by simply uploading the ontology and
selecting various metrics.
Out of the six papers, none of the tools are available for
reuse or public consumption, mainly due to their experimental nature. We also have examined, for each evaluation tool, how metrics are used to measure the quality of
the ontology. We assume that metrics provide a means to
automate and communicate quality.
While experimental tools have been proposed over the
last 10 years, none of them are available in the form of a
distributable application or are hosted on a publicly available platform that can promote widespread evaluation.
With the availability of widespread and documented evaluation, the ontology community can produce validated
and verified ontological knowledge bases with little errors
or inconsistencies. OntoKeeper is intended to produce a
tool that is available for broad usage for any domain. Most
of our work is in the biomedical sphere, but we foresee
OntoKeeper being applicable for a wide-range of domains
other than biomedicine. Also, OntoKeeper utilizes a metric suite that is easy to use and easy to interpret, so that
any ontologist could make the necessary improvements
for their ontology. The following sections will describe the
background theory behind the metric suite.
Our review of the papers revealed some important
observations. We realized how evaluation tools for
ontologies were harnessing a set of metrics that were
developed by the authors or adapted some pre-existing
metrics that has been published, as this would help to
quantify the evaluation that can be quickly calculated by
a machine. Unique among the other tools was Jimborean
and Gorza [12] incorporating some support for competency questions which is sometimes used as a method
to evaluate an ontology [19]. Ranking ontologies was a
common theme among the papers as it would serve as
a benchmark to decipher metric scoring. Of interest,
Burton-Jones and colleagues’ semiotic metric was mentioned in two of the six papers. Also, one of the tools
mentioned in [8] relied on semiotic theory for ontology
Ontologies and semiotics
Ontologies are “a formal, explicit specification of a shared
conceptualization” [20]. As electronic artifacts, they represent concepts and the semantic relationships that connect them to imbue meaning, context, and reasoning for
machines to consume and process. For machines to consume and process ontologies a machine-readable syntax,
such as RDF (Resource Description Framework) [21] or
OWL2 (Web Ontology Language, v2) [22], are used to
serialize the knowledge.
Semiotics, specifically semiotic theory, is the study of
the interpretations of signs and symbols [23, 24]. Semiotics is organized in three branches–syntactic, semantic,
and pragmatic [25]. Building on classic semiotic theory,
several authors have recently presented arguments for the
application of semiotic theory to contemporary initiatives.
For example, Price noted work that demonstrated the
value of applying semiotics to understanding information
systems and systems analysis, evaluating data model quality, and to evaluating information quality [26]. Applying
semiotics theory to contemporary modeling and simulation, Tolk and colleagues defined “syntactical entropy
that measures the variety of data representation, semantic
entropy that measures the variety of data interpretation,
and pragmatic entropy that measures the variety of data
utilization” [27].
To some researchers, ontologies are also semiotic
artifacts [13, 28–30]. Echoing Dividino and colleagues,
ontologies are symbolic and meaningful representations
of a domain space (semantics), constructed in a graphbased format (syntactical). The utility and understanding
of the ontology hinges on social, cultural and environmental factors (pragmatics). Because of the semiotic nature of
ontologies, it stands to reason that one way to comprehensively evaluate an ontology is to use evaluation standards
rooted in semiotic theory. OntoKeeper utilizes and builds
upon the Burton-Jones metric suite for semiotic-based
ontology evaluation.
Burton-Jones, et al. semiotic metric suite
Over a decade ago, Burton-Jones, et al. developed a set of
metrics that harnessed semiotic theory to grade an ontology quality [13] (See Table 2). The benefits of this suite
is that 1) it is independent of domain specificity or ontology language, and 2) it is a simple and thorough evaluation
system. The metric suite is composed of four branches syntactic, semantic, pragmatic, and social. The first three
are attributed to branches of semiotics while the fourth
is a layer introduced by the authors. The syntactic score
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Table 2 Constituents of the semiotic metric suite
Metric
Sub-Metric
syntactic (S)
lawfulness (SL)
richness (SR)
semantic (E)
Page 4 of 18
specific use case). The social score consists of an authority
score, based on the number of links to the subject ontology, and the history score, based on access to the ontology.
We will discuss the specifics on how to calculate these
scores in a subsequent section.
Method
interpertability (EI)
OntoKeeper
consistency (EC)
OntoKeeper is a Java-based web application that analyzes
ontology files (.owl or .rdf ) using semiotic metrics from
Burton-Jones and colleagues. OntoKeeper is the latest
upgraded evolution of the author’s previous tool, SEMS
[2]. The current version has refined the metric calculation,
improves on interface and functionality, and incorporates
natural language generation feature, which is harnessed
from the Hootation API library [31].
clarity (EA)
pragmatic (P)
comprehensiveness (PO)
accuracy (PU)
relevancy (PR)
social (O)
authority (OT)
history (OH)
concerns the machine-readability of the ontology artifact,
specifically asking if the ontology “can be read” [13]. The
semantic score assesses the appropriateness of the entities’ labels within the ontology, or if the ontology “can be
understood” [13]. The pragmatic score pertains to measuring the utility of the ontology, or if the ontology is
“useful” [13]. Finally, the social score measures the ontology status among the community of ontologies (i.e. “Can
it be trusted?” [13]). For some ontologies, particularly,
those that are new or in-development, the social score
may be neglected. The overall quality score is a composite of syntactic, semantic, pragmatic, and social. A high
overall score generally means better. Later, we will discuss
in Use Cases, how this overall quality score can be compared to an average of the overall quality score from a set
of ontologies to determine its quality. Furthermore, as we
will discuss later (See Use Cases), some of the scores may
be adjusted by modifying their weights or removing them
from the metric suite.
Each of the four core metrics are decomposable into
several sub-metrics. With syntactic, there is lawfulness
and richness. Lawfulness measures adherence to syntactic rules of the ontology profile, while richness measures the amount of ontology-related features (ClassAssertions, DomainObjectProperties, etc.). The semantic
score involves interpertability, consistency, and clarity.
Clarity discerns how ambiguous the term labels may be,
while consistency calculates the ratio of inconsistent use
of terms–for example, using the term “male” as a class
and repeating it as a label for an instance. Interpretability
involves calculating whether the label is meaningful. Pragmatic entails comprehensiveness (a measure of the ontology’s domain scope), accuracy (veracity of the knowledge
embedded in the ontology), and relevancy (fulfillment of a
Application architecture
OntoKeeper was developed with the Vaadin Java web
framework (v7.7), along with various third-party API
components to provide specific functionality. OntoKeeper
also utilized a PostgreSQL database (v9.5.8) to store basic
application data and natural language statements. The
test version of OntoKeeper was deployed on a Jetty web
server (v9), hosted on an Ubuntu v16.04.3 LTS machine
(4GB RAM and dual CPU cores). OntoKeeper was primarily developed by one of us (MA) and was evolved
from the previous iteration of the tool mentioned in [2].
Figure 1 briefly summarizes the main components, and
their interaction with each other.
In the figure (Fig. 1) SEMS Service was a port of the
code from the original SEMS web application. It was partitioned by other components (Syntactic, Semantic and
Pragmatic components)that were responsible for calculating each of the metrics and sub-metrics, except for the
Social component module which is inactive.
Each of the metrics modules heavily relied on the Ontology Service component to parse meta-data and label
information from the ontology. The Ontology Service
interfaced with either an ontology artifact that has been
uploaded (Ontology file) or an ontology that has already
previously uploaded and stored in the database through
the Database Service. The Ontology Service also required
the OWL-API to access functionality for the parsing of an
ontology artifact.
In addition, the WordNet Service relied on the Ontology Service to access the label-related information of the
ontology. The WordNet Service utilized the MIT JWI, a
Java WordNet interface, that queries a WordNet database
[32]. WordNet Service primarily provided the word sense
information for each token from the labels.
The NLG Service was primarily responsible for the
natural language generation of the ontology. It accessed
the ontology either through the database (Database
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 5 of 18
Fig. 1 General UML component architecture of OntoKeeper. Grayed components indicate inactive
Service) or the uploaded ontology (Ontology Service).
Also through the Database Service it saved the natural
language sentences for each of the triples.
Aside from providing services to other components, the
Database Service was also leveraged by the application
(OntoKeeper UI component) where the tool stored and
retrieved application data to function.
Application navigation
Regarding interface design, we aimed to refine the tool
to be simple and easy to navigate throughout the various
scores, as well as minimizing the amount of information
to avoid cognitive overload. The tool was also designed to
be responsive to various devices, so in the later part of this
section, we present screenshots of the mobile version. The
following section describes the interface starting from the
login screen to a final screen that showed the final quality score. Also, in this section, we introduce the interface
from which external domain experts will access to judge
the accuracy of the knowledge embedded in the ontology.
After navigating to the URL address of the application,
the ontologist user will encounter the login screen and will
be prompted for their username and password (Fig. 2).
Figure 3 is the next screen the user views after successfully logging in. The entire application has a visible
sidebar menu that allowes the user to navigate between
the different sections of the application. The Introduction
screen which greets the user after login, has three tabs.
The first tab is a short video demonstrating a quick usecase on how to use the tool. The second tab permits the
user to change their username or password, and the third
tab shows the saved snapshots of scores from previous
sessions.
The Configuration screen (Fig. 4) is where the user
starts the process in attaining scores for their ontology or
importing an existing ontology the user has uploaded previously. Any ontology file uploaded will be saved into the
database automatically for later retrieval. The first panel
has two tabs, Upload Ontology and Select an Ontology.
The former is where the user will choose the ontology
from their machine and upload it to the server. The other
tab will present the user with a list of ontologies that
the user has previously uploaded. The user can select the
ontology and click Import to load the ontology. Currently,
we advise the users to merge their ontology (via the Protégé editor) if it imports external ontologies, because the
system will calculate the scores based only on what is
local to the file and will not follow OWL imports. With
a merged ontology, the entities and properties from the
imports will be considered into the scoring. In the future,
we plan on adding support to automatically import the
external ontologies.
The other panels on the Configuration screen includes
the Ontology Status panel that indicate whether the ontology has been loaded, with the option to remove the
ontology from the session. The Excluded Aspects panel
allows the users to exclude scores from the four aspects
of syntactic, semantic, pragmatic, and social. The Parsing Options panel gives control to users on how to parse
non-alphanumeric characters. By default all the options –
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 6 of 18
Fig. 2 Splash screen
fixing camel cased labels, removing determiners, brackets,
underscores, and dashes – are selected.
After the ontology has been loaded and the session configured, the next screen is the Processing screen (Fig. 5).
The labels of the ontology is outputted and displayed in
the grid after the Process button has been clicked. The
grid displays the original label and the post-processed
label based on the configuration. Also, the number of
word senses the label has is based on the WordNet
database. For labels with multiple tokens, the word senses
are added to form an accumulated word sense total.
Syntactic calculation The syntactic score (Eq. 1) is composed of the lawfulness (SL) and richness scores (SR).
The lawfulness is calculated by attaining the total number of axioms (logical and non-logical axioms), which
are derived from OWL-API [17]. By instantiating the
OWL2DLProfile class with the OWL-API, we also collected the number of violations. Using that count we
divide it by the total number of axioms, resulting in the
lawfulness score.
S = ws1 ∗ SL + ws2 ∗ SR
SL = slv /AX
SR = srfeatures /srtotalfeatures
let AX represent all logical and non − logical axioms
(1)
For richness, we used the OWL-API to determine the
number of features of the language used in the ontology
being evaluated. This was then divided by the number of
possible features in the ontology, which for OWL is 39.
This quotient provided us with the richness score.
Figure 6 shows the Syntactic screen that displays the
scores related to the syntactic measures. The two tabs
relate to the syntactic measures of lawfulness and richness.
Each of these panels displays the scores for these two measures along with a simple explanation of the scores. The
other panel contains slider widgets that allow the user to
diminish or strengthen one of the scores.
Semantic calculation Semantic score (Eq. 2) relies on
OWL-API, and WordNet [33] to derive the number of
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 7 of 18
Fig. 3 Introduction screen
word senses that each word has. For the interpretability
score (EI), we took the number of unique words from all
of the labels that are parsed from the ontology. For each
unique word, we used WordNet to discover if the word has
at least one word sense, and recorded the total. Using that
total, we divided it by the total number of unique words in
the ontology. The resulting value is then subtracted from
1 to provide us the interpretability score.
E = we1 ∗ EI + we2 ∗ EC + we3 ∗ EA
EI = 1 − (tsense /t)
(tavg_senses )
EA = 1 −
t
EC = 1 − (d/t)
let t = unique tokens ⊂ ontology labels,
tsense = total tokens with one word sense,
tsenses = total sense per token,
tavg_sense = average sense per token,
d = non − unique tokens ⊂ ontology labels
(2)
With clarity (EA), we utilized the average number of
word senses per unique word, and divide that value with
the total number of unique words. With that value, we
subtracted that from 1 to obtain the clarity score.
Consistency score (EC) is calculated by counting the
number of duplicate words and dividing that figure with
the total number of unique words. That value is subtracted
from 1 to attain the consistency score.
Similarly, the Semantic screen (Fig. 7) also has the same
widget to modulate the three semantic scores of interpertability, consistency, and clarity. There are also three
tabs for each of those scores, with an explanation of the
scores.
Pragmatic calculation We used the OWL-API to collect the number of classes, instances, data properties,
and object properties. The total number of all four of
these elements amounted to the total number of elements used to calculate the comprehensiveness score (PO)
for the pragmatic score (P). Also needed was an average number of number of elements (classes, instances,
data properties, and object properties) from a group or
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Fig. 4 Configuration screen
Fig. 5 Process screen
Page 8 of 18
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 9 of 18
Fig. 6 Syntactic screen
library of representative ontologies. The total number of
elements from the ontology being assessed, was divided
by the aforementioned average number of elements. For
example, if we have a food-related ontology, we would
require the average of classes, instances, and properties
from similar food-related ontologies that are available,
or we could attain the average from a general ontology repository/library, like NCBO BioPortal, if there is
a scarcity of similar ontologies. All in all, the numeric
values results in the comprehensiveness score of the
ontology.
The accuracy score (PU) relies on the Hootation API
(See “Hootation” section) and external experts. All of the
logical axioms from the ontology were translated into natural language. The external domain experts assessed if
each statement was true or false. The number of true
statements was collected and averaged, and the final value
produced the accuracy score.
P = wp1 ∗ PO + wp2 ∗ PU + wp3 ∗ PR
PO = CIDOn /CIDOaverage
PU = AXtruen /AXlogicaln
let CIDOn = {classesn ∪ instancesn ∪ data propertiesn
∪ object propertiesn , }
CIDOaverage = average from set or library of ontologies,
AXlogical_axioms ⊂ AX, logical axioms from all axioms,
AXhuman ≈ AXlogical_axiom ,
natural language translation of axioms,
AXtruen ,number of true AXhuman ,
AXlogicaln , number of AXlogical_axioms
(3)
Relevancy score (PR) is not supported in OntoKeeper,
as it a score that is specific to a use-case defined by
the evaluator. For example, an evaluator may create a set
of competency questions and calculate the percentage of
adherence for the questions to determine the relevancy
score. Relevancy is understood as being a score to measure
performance of a task, specifically a user-defined task.
Most of the calculations, are automated, but the pragmatic scoring is a bit more involved. Figure 8 shows the
Pragmatic screen, and like the previous, it also has slider
widgets to control the influence of the pragmatic scores.
It has three tabs for each of the pragmatic scores. The first
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 10 of 18
Fig. 7 Semantic screen
tab for comprehensiveness, displays its score and has a text
field for the user to input the average number of ontology
elements (classes, properties, and instances). This average
value may vary depending on the number of ontologies
that are being compared. In our previous study [34], we
noted that this number may vary (i.e. 1,277,993 for NCBO
Bioportal, 169,862 for a set of drug ontologies). In [13],
Burton-Jones, et al. used 500, but over the last decade the
size of ontologies have greatly increased, and the comprehensiveness score may elicit a value greater than 1. What
we have performed, and what is recommended by [13] is
to collect a set of ontologies that are of a similar domain
and record the total number of elements to input.
The second tab for the Pragmatic screen is more
involved. Like all the other tabs, it displays information
about the sub-score, but it also has functionality to enlist
volunteer domain experts to assess the truthfulness of
the ontology. The Preview Statements button allows the
user to view the list of natural language statements that
are from the ontology’s axioms (See Fig. 9). This Review
screen will be the same UI as what the enlisted domain
experts will experience (See Fig. 10). From Fig. 11, there
is also a panel labeled Subject Matter Volunteers. In this
widget, the user adds the domain experts to be sent an
invitation to examine the user’s ontology. From this panel,
the user can remind the volunteers to participate and also
view their private link to access their unique grid to review
the ontology (Fig. 10). In the review, the volunteer can
indicate whether the statement is true or false, and add
any notes.
Hootation Hootation API is a Java library based
on natural language generation (NLG) components
from the Agile Knowledge Engineering and Semantic Web Group’s semantic web application for generating quiz questions [35]. At the time of our past
study [31], only 14 logical axiom types were supported,
but currently Hootation supports 25 logical axioms
types.
A few of the metrics provided through OntoKeeper
requires external participants and resources. One such
metric (accuracy) needs domain experts to assess the
veracity of the triples in the ontology. Most domain
experts are not familiar with ontology languages or tools.
Exporting the logical axioms to human readable language
would enable accessibility for domain experts with little
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Fig. 8 Pragmatic screen
Fig. 9 Review of the natural language statements from the axioms
Page 11 of 18
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 12 of 18
Fig. 10 Expert view of the natural language generated statements from an invite link. Same grid UI as Fig. 9
ontology experience, even though the knowledge triples
are expressed in descriptive logic.
Social calculation Also, due to technical limitations, the
current iteration of OntoKeeper does not calculate the
social score (Eq. 4). However, the score is comprised of the
history score (OH) and authority score (OT). The authority score is based on an average times that ontology has
been accessed within a library of ontologies, and the history score is calculated by the number of ontologies of a
certain library that links to the ontology.
O = wo1 ∗ OT + wo2 ∗ OH
(4)
Overall Quality Calculation The overall quality (Eq. 5)
is a composite score of semantic (E), syntactic (S), pragmatic (P), and social score (O). Each score is modulated
with weights (wqn ) to balance their degree of strength.
In a previous publication, we noted how the weights can
be leveraged to provide a more accurate composite score
among similar ontologies [34].
Q = wq1 ∗ S + wq2 ∗ E + wq3 ∗ P + wq4 ∗ O
(5)
The final screen of importance is the Summary section
(Fig. 12). In this screen, the overall quality score is displayed along with some visualizations to indicate scores
for each of the quality aspects. As noted earlier, social
score is not supported and thus grayed out on the UI. Similar to the sub-score screens, the user has the option to
adjust the strengths of each score. In Fig. 12, for demonstration purposes, the syntactic score is weighted at 0.15,
semantic is weighted at 0.51, and pragmatic at 0.33. The
final scoring of the session can be saved for archiving using
the Save Snapshot panel.
Of some worth, the application is also usable through
mobile device by way of a responsive design. Figures 13
and 14 show the login screen and the Pragmatic screen
rendered from an Android smartphone. With a streamlined interface and adaption to various screen sizes, we
foresee that in the future this application could be usable
for mobile users. Further refinement of the interface is still
needed and the possibility of an ontology artifact residing
on someone’s smartphone is remote. For the usability testing, as we will introduce in the next section, our evaluators
utilized their desktop instead of their portable devices.
Usability evaluation
Five of the co-authors (CT, YH, CL, DW, and FM), who
have published research and development experience with
ontologies, participated in assessing OntoKeeper independently. None of the five co-authors were involved in
the development of OntoKeeper. Each participant was
furnished with a username and password to login, and
each participant did not receive any guidance. Each user
were left to their own to operate the tool by uploading an
ontology of their choice and explore the tool without any
intervention. After reviewing and testing the tool, each
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 13 of 18
Fig. 11 Accuracy tab screen
participant completed a survey using the System Usability Scale (SUS) [36, 37] to appraise the tool. The SUS
instrument is a simple 10-item survey using a Likert scale
for each item (1=strongly disagreed, 5=strongly agreed).
Additionally, SUS is known for its reliability with a small
sample [38]. The scores were compiled and discussed in
the next section. Lastly, the survey provided free text space
to allow further comments that are not covered by the
survey.
Results
Usability results
For each participant, we collected the results of the System
Usability Scale (SUS) survey from all five of the participants and calculated the scores [37] (See Table 3). The
average SUS score was 93.5, and participant scores ranged
from 87.5 to 100. According to usability studies, the baseline score for average usability using the SUS scale is 68
[38]. With a score in the high 80s and above, it reasonable
to assume that participants agreed that OntoKeeper had
very high attributes in usability. Usability experts would
place the score in the top 96–100 percentile with a grade
of “A+” [39].
Also, we calculated the standard deviation for each of
the items to verify any divergence of opinion. For eight
items, there appeared to be uniformity in opinions based
on the standard deviation values. However, for two items
there appear to be some variability:
• I think that I would need the support of a technical
person to be able to use this system.
• I needed to learn a lot of things before I could get
going with this system.
We deduced that these two items are similar in nature,
addressing the need for guidance or simple learning material. Noted earlier, no assistance was given, and participants independently operated the tool on their own volition. The Introduction screen featured a video demo, but
we did not ascertain whether the participants watched the
video. Although, one user noted that a more detailed video
would be helpful.
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 14 of 18
Fig. 12 Summary screen
Each participant had an opportunity to provide feedback (positive, negative) of his or her experience using
OntoKeeper. Some of the feedback hinted at suggestions for improvement. Regarding positive feedback, users
noted the ease of use of the application and the accessibility of the tool for ontologists of varying expertise.
Users suggested that the overall quality score should be
persistent throughout the session without having to navigate between tabs. Users also noted challenges with the
grid that displays the natural language statements, where
there may be numerous translated axioms. This could
potentially lead to difficulties for domain experts who are
tasked to review the grid of natural language statements.
Ideas such as a filter or suggesting an alternative way to
display natural language statements might solve this issue.
Some users indicated difficulty comprehending specific
parts of the application. One user specified providing
more concise instructions on the interface. This issue
might harken back to the need for better learning material
or an interface with more guidance involved.
Use cases
We have utilized the semiotic metric suite in two previous studies. In one study, we employed the semiotic-based
evaluation system on a set of NCBO ontologies and drug
ontologies. The goal was how to effectively use the metric suite to provide meaningful metrics. The other study
involved the use of natural language generation, a subfield of natural language processing, where data is transformed to free text for human understanding of the data.
The goal of using natural language generation is to provide better facilities for non-technical domain experts
to assess ontologies. These previous works offered were
incorporated and consolidated into the OntoKeeper platform to automate the ontology evaluation process for
assessing biomedical ontologies in the form of a software
application.
Utilizing semiotic metrics for drug ontology evaluation
We had utilized this metric suite to evaluate a group
of NCBO drug ontologies [34]. In that study we were
posed with the question on how to use this metric suite
to precisely evaluate a group of drug ontologies. From a
random sample of 64 ontologies (from September 2015
among most frequently visited) from the NCBO Bioportal, we calculated each ontology using the metric, and
recorded the mean for each of the scores and the overall quality score for the sample. This provided us a “gold
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 15 of 18
Fig. 13 OntoKeeper rendered from smartphone device
standard” to evaluate a group of drug ontologies that
included RxNORM [40], Veterans Health Administration National Drug File (VANDF) [41], The Drug Ontology (DRON) [42], The Drug-Drug Interactions Ontology
(DINTO) [43], Vaccine Ontology (VO) [44], and Pharmacovigilance Ontology (PVOnto) [45].
We also postulated if one were to create a new drug
ontology or compare an existing drug ontology with other
drug ontologies, that it would be suitable to compare
ontologies from a similar domain or sort. We recorded
an aggregated score of drug ontologies to create our
gold standard comparison but modified the weights based
on some observed strengths and weakness of the drug
ontologies. The end result yielded modulated weights for
each of the scores, specific for drug ontologies.
ontology experience and ontologists. NLG is particularly
useful for allowing domain experts to assess the veracity of
the knowledge that is encoded in an ontology, considering
that the coded axioms are translated into natural language
free text.
We explored the feasibility of using NLG for the task
of knowledge validation using an NLG engine for OWL
2 [31]. We produced natural language statements from
logical axioms from three ontologies – “People” ontology
[46], Time Event Ontology [47], and Informed Consent
Ontology [48]. We had evaluators familiar with each of the
ontologies to examine the quality of the natural language
output [31]. Overall, the evaluators indicated that most of
the output provided clear natural language sentences for
each of the triples.
Natural language generation for ontology evaluation
Discussion
Natural language generation (NLG) offers a feasible
method to close the gap between domain experts with no
Two items from the survey indicated some issue with
learning some of the features of OntoKeeper in order
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 16 of 18
Fig. 14 OntoKeeper rendered from smartphone device
to get started. We may need to devise some alternatives
besides the introductory video that could help users better operate the tool quickly - integrated user-generated
wiki, guided prompts, etc. While the System Usability
Scale has some reliability a with small sample users [38],
additional evaluators are needed to provide a significant
assessment. Aside from the usability, we also acknowledged some technical limitations. As we discussed before,
social score is a feature that is unavailable for a variety of
reasons. Specifically we have to determine what are the
ontologies that are linking to the subject ontology. However, in light of the difficulty in automating this feature,
we could implement some manual input to allow the user
to add the number, until a feasible automated solution is
available. In addition, we also have to determine the number of times the ontology has been accessed. Similarly,
with the comprehensiveness tab, the user has to have a priori knowledge of the number of ontology elements from
domain-relevant library of ontologies. Overall, there may
be a need to have a stored library or database of ontologies
that OntoKeeper can access and gather some of the data
needed for the aforementioned scores.
OntoKeeper uses the Hootation API that translates the
ontology’s axioms to natural language statements. However, one user noted that the URI (Uniform Resource
Identifier) for an entity appeared in a sentence instead of
the label. This might be due to an unmerged ontology or
some other issue we need to investigate. If it is the former,
OntoKeeper may need to have some implementation in
place that will import the external ontologies and merge
them to the user’s ontology.
Overall, some of these limitations can be surpassed
with modifications in the design, further development of
the tool, and additional hardware resources. Currently,
the tool resides on our development server. Our eventual goal is provide this tool as a service for knowledge
engineers to utilize. One idea to extend OntoKeeper is
to allow users to publish their scores publicly through
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Page 17 of 18
Table 3 p notation represent individual participants ratings
SUS Items
p1
p2
p3
p4
p5
µ(σ )
I think that I would like to use this system frequently
4
5
5
5
5
4.8 (0.45)
I found the system unnecessarily complex
1
1
1
1
1
1 (0)
I thought the system was easy to use.
5
5
3
5
5
4.6 (0.89)
I think that I would need the support of a technical
person to be able to use this system.
2
5
1
1
1
2 (1.73)
I found the various functions in this system were well
integrated.
5
5
5
5
5
5 (0)
I thought there was too much inconsistency in this
system.
1
1
1
1
1
1 (0)
I would imagine that most people would learn to use
this system very quickly
5
5
5
4
5
4.8 (0.45)
I found the system very cumbersome to use.
1
1
1
1
1
1 (0)
I felt very confident using the system.
5
5
5
5
5
5 (0)
I needed to learn a lot of things before I could get
going with this system.
2
1
4
1
1
1.8 (1.30)
SUS Calculated Score
92.5
90
87.5
97.5
100
Values in parentheses are the standard deviation.Values derived from Likert scale (1=strongly disagreed, 5=strongly agreed)
OntoKeeper. We envision a public portion of the tool that
lists ontologies that have been tested, along with supplementary data like the natural language statements. This
might be beneficial for researchers to document the evolution of their ontologies or publically announce the quality
of their knowledge-base. Other ideas include having an
open API from the server to allow external tools, like Protégé, to leverage OntoKeeper’s backend services. We also
alluded to mobile, and we may investigate the feasibility of
mobile technologies for ontology evaluation.
Conclusion
Our review of the literature for automated ontology evaluation tools show the lack of software that could be used by
ontologists and researchers to measure the overall quality
of ontologies. We presented OntoKeeper as the latest iteration of our contribution towards ontology evaluation and
quality assurance. OntoKeeper is a web-based application
that permits ontologist users to grade the quality of their
ontology based on semiotic measures devised by BurtonJones and colleagues. The system was designed for public
use, and we intend to make this service public and refine
specific aspects of the metric suite. With five evaluators,
we surmised that the system is deemed to be usable with
a SUS score above the baseline. Nonetheless, there is further work needed to enhance the usefulness of the tool for
ontologist users.
Acknowledgements
Not applicable.
About this supplement
This article has been published as part of BMC Medical Informatics and Decision
Making Volume 19 Supplement 4, 2019: Selected articles from the Third
International Workshop on Semantics-Powered Data Analytics (SEPDA 2018). The
full contents of the supplement are available online at https://
bmcmedinformdecismak.biomedcentral.com/articles/supplements/volume19-supplement-4.
Authors’ contributions
MA developed the software tool and the first manuscript draft. FM, CL, DW, YH,
and CT collected the usability data. FM, CL, DW, YH, MH, and CT provided
revisions for the subsequent manuscript drafts. All authors read and approved
the final manuscript.
Funding
Publication costs and research are funded by the UTHealth Innovation for
Cancer Prevention Research Training Program (Cancer Prevention and
Research Institute of Texas grant #RP160015), National Institutes of Health
under Award Numbers U01 HG009454, and R01 LM011829.
Availability of data and materials
Not applicable.
Ethics approval and consent to participate
Not applicable.
Consent for publication
Not applicable.
Competing interests
The authors declare that they have no competing interests.
Author details
Abbreviations
ACM: Association of Computing Machinery; AHP: Analytical Hierarchical
Process; IEEE: Institute of Electrical and Electronics Engineers; NCBO: National
Center for Biomedical Ontologies; NLG: Natural language generation; OWL2:
Web Ontology Language, version 2; RDF: Resource Description Framework;
SUS: System Usability Scale; URI: Uniform Resource Identifier
1 School of Biomedical Informatics, The University of Texas Health Science
Center at Houston, 7000 Fannin Street, Suite 600, 77030 Houston, TX, USA.
2 Department of Systems, Populations and Leadership, University of Michigan
School of Nursing, 426 N. Ingalls St, 48109 Ann Arbor, MI, USA. 3 Arnold School
of Public Health, University of South Carolina, 29208 Columbia, SC, USA.
4 University of Texas, 78712 Austin, TX, USA. 5 Center for Computational
Amith et al. BMC Medical Informatics and Decision Making 2019, 19(Suppl 4):152
Medicine & Bioinformatics, University of Michigan Medical School, Room 2017,
Palmer Commons 100 Washtenaw Avenue, 48109 Ann Arbor, MI, USA.
Published: 8 August 2019
References
1. Abele A, McCrae J. The Linking Open Data cloud diagram. 2017. http://
lod-cloud.net. Accessed 10 June 2019.
2. Amith M, Tao C. A web application towards semiotic-based evaluation of
biomedical ontologies. In: International Workshop on Biomedical Data Mining,
Modeling, and Semantic Integration: A Promising Approach to Solving
Unmet Medical Needs, BDM2I 2015-co-located with the 14th International
Semantic Web Conference, ISWC 2015. Bethlehem: CEUR-WS; 2015.
3. Brank J, Grobelnik M, Mladenić D. A survey of ontology evaluation
techniques. In: Proceedings of the Conference on Data Mining and Data
Warehouses (SiKDD 2005). Ljubljana; 2005.
4. Amith M, He Z, Bian J, Lossio-Ventura JA, Tao C. Assessing the practice
of biomedical ontology evaluation: Gaps and opportunities. J Biomed
Inform. 2018;80:1–13.
5. Tartir S, Arpinar IB. Ontologyevaluation and ranking using ontoqa. In: International
Conference On Semantic Computing. Irvine: IEEE; 2007. p. 185–92.
6. Tartir S, Arpinar IB, Moore M, Sheth AP, Aleman-Meza B. Ontoqa:
Metric-based ontology quality analysis. In: IEEE ICDM Workshop on
Knowledge Acquisition from Distributed, Autonomous, Semantically
Heterogeneous Data and Knowledge Sources. New Orleans: IEEE; 2005.
7. Jianliang X, Xiaowei M. A web-based ontology evaluation system. In:
International Conference On Advanced Language Processing and Web
Information Technology. Liaoning: IEEE; 2008. p. 104–7.
8. Aruna T, Saranya K, Bhandari C. A survey on ontology evaluation tools. In:
2011 International Conference On Process Automation, Control and
Computing. Coimbatore: IEEE; 2011. p. 1–5.
9. Zhu H, Liu D, Bayley I, Aldea A, Yang Y, Chen Y. Quality model and
metrics of ontology for semantic descriptions of web services. Tsinghua
Sci Technol. 2017;22(3):254–72.
10. Liu D, Yang Y, Chen Y, Zhu H, Bayley I, Aldea A. Evaluating the
ontological semantic description of web services generated from
algebraic specifications. In: 2016 Symposium On Service-Oriented System
Engineering. Irvine: IEEE; 2016. p. 211–20.
11. Groza A, Dragoste I, Sincai I, Jimborean I, Moraru V. An ontology
selection and ranking system based on the analytic hierarchy process. In:
2014 16th International Symposium On Symbolic and Numeric
Algorithms for Scientific Computing. Timisoara: IEEE; 2014. p. 293–300.
12. Jimborean I, Groza A. Ranking ontologies in the ontology building
competition boc 2014. In: 2014 International Conference On
Communication Problem-Solving. Beijing: IEEE; 2014. p. 75–82.
13. Burton-Jones A, Storey VC, Sugumaran V, Ahluwalia P. A semiotic
metrics suite for assessing the quality of ontologies. Data Knowl Eng.
2005;55(1):84–102.
14. Saaty TL. Decision making with the analytic hierarchy process. Int J Serv
Sci. 2008;1(1):83–98.
15. Alani H, Brewster C, Shadbolt N. Ranking ontologies with aktiverank. In:
International Semantic Web Conference. Berlin: Springer; 2006. p. 1–15.
16. Yu W, Li Q, Chen J, Cao J. Os_rank: structure analysis for ontology
ranking. In: 23rd International Conference on Data Engineering. Istanbul:
IEEE; 2007. p. 339–46.
17. Horridge M, Bechhofer S. The owl api: A java api for owl ontologies.
Semantic Web. 2011;2(1):11–21.
18. Apache Software Foundation. Apache Jena. https://jena.apache.org.
Accessed 10 June 2019.
19. Gómez-Pérez A. Ontology evaluation. Berlin Heidelberg: Springer; 2004,
pp. 251–73.
20. Studer R, Benjamins VR, Fensel D. Knowledge engineering: principles
and methods. Data Knowl Eng. 1998;25(1-2):161–97.
21. Cyganiak R, Wood D, Lanthaler M. Rdf 1.1 concepts and abstract syntax.
W3c Recomm. 2014;25(02). https://www.w3.org/TR/rdf11-concepts/.
Accessed 8 July 2019.
22. Consortium WWW, et al. OWL 2 Web Ontology Language Document
Overview (Second Edition). 2012. https://www.w3.org/TR/owl2overview/. Accessed 4 July 2019.
23. De Saussure F. Course in General Linguistics. New York: Columbia
University Press; 2011.
Page 18 of 18
24. Culler J. The Pursuit of Signs. London: Routledge; 2005.
25. Morris C. Foundations of the Theory of Signs, vol 1. Chicago: University of
Chicago Press; 1938.
26. Price R, Shanks G. A semiotic information quality framework:
development and comparative analysis. In: Enacting Research Methods in
Information Systems. Cham: Palgrave Macmillan; 2016. p. 219–50.
27. Tolk A, Diallo SY, Padilla JJ. Semiotics, entropy, and interoperability of
simulation systems: mathematical foundations of m&s standardization.
In: Proceedings of the Winter Simulation Conference. Berlin: 2012 Winter
Simulation Conference; 2012. p. 243.
28. Dividino RQ, Romanelli M, Sonntag D, et al. Semiotic-based ontology
evaluation tool (s-ontoeval). In: Language Resources and Evaluation.
Marrakech: Morocco; 2008.
29. Gangemi A, Catenacci C, Ciaramita M, Lehmann J. Qood grid: A
metaontology-based framework for ontology evaluation and selection.
In: 4th International EON Workshop located at the 15th International
World Wide Web Conference. Edinburgh, United Kingdom; 2006.
30. Stamper R, Liu K, Hafkamp M, Ades Y. Understanding the roles of signs
and norms in organizations-a semiotic approach to information systems
design. Behav Inf Technol. 2000;19(1):15–27.
31. Amith M, Manion F, Harris M, Zhang Y, Xu H, Tao C. Expressing
biomedical ontologies in natural language for expert evaluation. Stud
Health Technol Inform. 2017;245:838–42.
32. Finlayson MA. Java libraries for accessing the princeton wordnet:
Comparison and evaluation. In: Proceedings of the 7th Global Wordnet
Conference, Tartu, Estonia, vol 137; 2014.
33. Miller GA. WordNet: a lexical database for English. Commun ACM.
1995;38(11):39–41.
34. Amith M, Tao C. Modulated evaluation metrics for drug-based
ontologies. J Biomed Semant. 2017;8(1):17. https://doi.org/10.1186/
s13326-017-0124-2.
35. Bühmann L, Usbeck R, Ngomo A-CN. ASSESS—Automatic
Self-Assessment Using Linked Data. In: International Semantic Web
Conference. Berlin: Springer; 2015. p. 76–89.
36. Brooke J. SUS-Aquickanddirtyusabilityscale. Usability Eval Ind. 1996;189(194):4–7.
37. Brooke J. SUS: a retrospective. J Usability Stud. 2013;8(2):29–40.
38. U.S. Department of Health & Human Services. System Usability Scale
(SUS). 2019. https://www.usability.gov/how-to-and-tools/methods/
system-usability-scale.html. Accessed 23 Apr 2019.
39. Sauro J, Lewis JR. Quantifying the User Experience: Practical Statistics for
User Research. Cambridge: Morgan Kaufmann; 2016.
40. Liu S, Ma W, Moore R, Ganesan V, Nelson S. Rxnorm: prescription for
electronic drug information exchange. IT Prof. 2005;7(5):17–23.
41. Veterans Health Administration National Drug File. Natl Cent Biomed
Ontol BioPortal. 2015. http://bioportal.bioontology.org/ontologies/
VANDF. Accessed 10 Apr 2017.
42. The Drug Ontology. Natl Cent Biomed Ontol BioPortal. 2015. http://
bioportal.bioontology.org/ontologies/DRON. Accessed 10 Apr 2017.
43. The Drug-Drug Interactions Ontology. Natl Center Biomed Ontol
BioPortal. 2016. http://bioportal.bioontology.org/ontologies/DINTO.
Accessed 10 Apr 2017.
44. Vaccine Ontology. Natl Center Biomed Ontol BioPortal. 2016. http://
bioportal.bioontology.org/ontologies/VO. Accessed 10 Apr 2017.
45. Pharmacovigilance Ontology. Natl Center Biomed Ontol Bioportal. 2011.
http://bioportal.bioontology.org/ontologies/PVONTO. Accessed10Apr2017.
46. Introduction to Ontologies with Protege. Department of Computer
Science and Software Engineering - Cal Poly, San Luis Obispo. 2011.
https://wiki.csc.calpoly.edu/OntologyTutorial/wiki/
IntroductionToOntologiesWithProtege. Accessed 26 Nov 2016.
47. Time Event Ontology. School of Biomedical Informatics, Ontology
Research Group, University of Texas Health Science Center. 2016. https://
sbmi.uth.edu/ontology/project/time-event-ontology.htm. Accessed 10
Apr 2019.
48. Lin Y, Harris MR, Manion FJ, et al. Development of a BFO-based Informed
Consent Ontology (ICO). Houston: Proceedings of the 5th International
Conference on Biomedical Ontology; 2014. http://ceurws.org/Vol-1327/
icbo2014_paper_54.pdf. Accessed 28 Apr 2016.
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in
published maps and institutional affiliations.