HOW CAN ONE EVALUATE A CONVERSATIONAL SOFTWARE AGENT FRAMEWORK?
Dr Kulvinder Panesar
School of Art, Design and Computer Science, York St John University, York, UK
Abstract
This paper presents a critical evaluation framework for a linguistically orientated conversational software agent
(CSA) (Panesar, 2017). The CSA prototype investigates the integration, intersection and interface of the
language, knowledge, and speech act constructions (SAC) based on a grammatical object (Nolan, 2014), and the
sub-model of belief, desires and intention (BDI) (Rao and Georgeff, 1995) and dialogue management (DM) for
natural language processing (NLP). A long-standing issue within NLP CSA systems is refining the accuracy of
interpretation to provide realistic dialogue to support the human-to-computer communication.
This prototype constitutes three phase models: (1) a linguistic model based on a functional linguistic theory –
Role and Reference Grammar (RRG) (Van Valin Jr, 2005); (2) Agent Cognitive Model with two inner models:
(a) knowledge representation model employing conceptual graphs serialised to Resource Description Framework
(RDF); (b) a planning model underpinned by BDI concepts (Wooldridge, 2013) and intentionality (Searle,
1983) and rational interaction (Cohen and Levesque, 1990); and (3) a dialogue model employing common
ground (Stalnaker, 2002).
The evaluation approach for this Java-based prototype and its phase models is a multi-approach driven by
grammatical testing (English language utterances), software engineering and agent practice. A set of evaluation
criteria are grouped per phase model, and the testing framework aims to test the interface, intersection and
integration of all phase models and their inner models. This multi-approach encompasses checking performance
both at internal processing, stages per model and post-implementation assessments of the goals of RRG, and
RRG based specifics tests.
The empirical evaluations demonstrate that the CSA is a proof-of-concept, demonstrating RRG’s fitness for
purpose for describing, and explaining phenomena, language processing and knowledge, and computational
adequacy. Contrastingly, evaluations identify the complexity of lower level computational mappings of NL –
agent to ontology with semantic gaps, and further addressed by a lexical bridging consideration (Panesar, 2017).
1
Introduction to the conversational agent
A conversational software agent (CSA) is a program that engages in conversation using natural language (NL)
dialogue with a human user. NL is the most easily understood knowledge representation for people, but
certainly not the best for computers because NL is inherently ambiguous, complex and dynamic. The challenge
here is for the system to encapsulate sufficient knowledge from the user’s question to present a grammatically
correct response. Here a discussion of knowledge and its role is very important. When a person hears or sees a
sentence, an individual will make full use of their knowledge and intelligence to understand it; additionally,
besides the grammar, knowledge about the words, the sentence context, and an understanding of the subject
matter is necessary. Here the computational model, must model this language understanding and the interactions
that combine grammar, semantics and reasoning.
Two main requirements for a CSA specified by Mao et al. (2012) and Lester et al. (2004) are the ability of
accurate natural language understanding (NLU) and the technical integration with an application. A CSA must
respond appropriately to the user’s utterance via three phases: (1) interpret the utterance, (2) determine the
actions (logic) that should be taken in response to the utterance (3) perform actions replying with text.
In the last decades, there has been a great evolution in the field of CSA enfolding three emerging trends of more
sophisticated natural language processing (NLP) via improved parsing techniques, humanising of agents
through language and their pervasiveness (Perez-Marin et al., 2011). CSAs are becoming prominent in the
marketplace where the consumer’s use and familiarisation of the smartphone technology and consumer support
agents, human-computer interaction, and ubiquitous computing is maturing, and is of great value.
1
A long-standing issue within NLP CSA systems is refining the accuracy of the interpretation of meaning to
provide a realistic dialogue to support the human-to-computer communication. The focus here is that NL is
conceived of as a functional system (Francois, 2014).
A functional mature grammar - Role and Reference Grammar (RRG) is deployed as a linguistic theory that can
adequately explain, describe and embed the communication-cognitive thinking in conversation, in a
computational form. With this in mind, we propose a language model, knowledge model, speech act
constructions (SAC) and belief, desires, and intentions (BDI) framework for a linguistically motivated text based
CSA namely LING-CSA. It will further investigate how language can be comprehended and produced, to gain a
deep understanding, and how it interfaces with knowledge. In order to use the RRG linking system and to
create an effective parser, RRG is re-organised to facilitate the innovative use of SAC as grammatical objects
(Nolan, 2014) and rich lexicon.
Figure 1 Re-organisation of RRG for the CSA
This paper presents a critical evaluation framework for LING-CSA (Panesar, 2017). It investigates the
evaluation of the integration, intersection and interface of the language, knowledge, and speech act constructions
(SAC) based on a grammatical object (Nolan, 2014), and the sub-model of belief, desires and intention (BDI)
(Rao and Georgeff, 1995) and dialogue management (DM) for natural language processing (NLP).
This objectives of this paper: (1) an account of the conceptual architecture of LING-CSA; (2) implementation
details; (3) an evaluation and testing of the CSA framework via a range of evaluation criteria and series of tests,
(4) a summary of the findings; (5) conclusions and recommendations.
2
Conceptual architecture of the conversational software agent
LING-CSA constitutes three phase models as illustrated in Figure 2 below.
Figure 2 Conceptual architecture of the LING-CSA (Panesar, 2017)
2
Figure 2 identifies: (1) RRG language model, (2) Agent Cognitive model (ACM) with two inner models of: (a)
knowledge model; (b) planning model; (3) Phase 3 – Agent Dialogue Model (ADM) where the DM is a common
component of Phases 1 and Phase 3 – due to the discourse referents of an utterance, and the need to create a
grammatically correct response.
2.1
RRG Language Model (Phase 1)
The LING-CSA will constitute a closed domain - dialogue on food and cooking, for its richness and universality.
A range of NLP tasks manipulated via the language model will include tokenisation, sentence splitting, part-ofspeech-tagging, morphological analysis, syntactic and semantic parsing. It is the DM that will assist with the SA
dialogue working with the syntactic parser to work effectively with the data sources (word list, lexicon, empty
speech acts constructions (SACs)). RRG manipulations exist for both simple and complex sentences, but the
LING-CSA is limited to the manipulation of the linking system of simple sentences (active or passive) using
transitive, intransitive and ditransitive verbs/auxiliary verbs with variable word order flexibility with SVO, and
modelled via speech acts (Searle, 1969) applied to cognitive and rational interaction analysis. RRG’s bidirectional linking algorithm and discourse-pragmatics interface will be mapped into completed SACs, invoking
the lexicon to supports all three LING-CSA phases, and internal BDI manipulations (Panesar, 2017).
Note: Logical structures have been partially expanded for readability in this paper
Table 1 - Snapshot of the lexicon - lexical entries and lexemes (data source 1)
LEXICAL
ENTRY
eat
POSTYPE
VERB
VERB TENSE/
ASPECT
PRS/ FUT
DEF
NO
GR
CASE
ANIM
HUM
DEF+/-
P
TYPE
3
SG
M/F
DNA
ANIM
HUM
PROG
DEF+/-
3
SG
M/F
DNA
ANIM
HUM
VBE
DNA
DEF+
DNA
DNA
DNA
DNA
DNA
DNA
LOGICAL STRUCTURE
(LS)
<tns:prs <do’(x, [eat’(x,y)] ) &
BECOME consumed’(y)] >>
<tns:fut <do’(x, [eat’(x,y)] ) &
BECOME consumed’(y) >>
<tns:prs <asp:prog <do’(x,
[eat’(x, y)] ) & BECOME
consumed’ (y)] >>>
be'(x,[pred'])
eating
VN
is
hungry
ADJ
DNA
DNA
DNA
DNA
M/F
DNA
ANIM
HUM
DNA
restaurant
N
DNA
DEF+/
DNA
SG/PL
DNA
DNA
DNA
DNA
DNA
Table -2 Overview of the RRG steps (Panesar, 2017)
Step
1
2
3
PHASE
(TRACKING) DISPLAY
PHASE 1 - RRG LANGUAGE MODEL
PHASE 1A - PREANALYSIS
PHASE 1B – SAC
SELECTION
PHASE 1C –
SYNTAX-TOSEMANTICS
Parsing, Sentence and word tokenisation,
check against the lexicon.
Construction selection computations
Display the utterance and various
internal representations.
Display construction selected
Display the verb selection and voice
topics
Display the LS and verb from the
lexicon
Display the argument value
Display the semantic representation
Map the operators to the Tense and
illocutionary force
Display the actor and undergoer as
part of the LS.
Display the additional information
4
PHASE 1E -
Verb selection and voice
Retrieve the LS and verb from the lexicon
Assignment of arguments to the LS
arguments
Semantic representation
Update the LS
Map the operator projection and focus
projections
Actor and undergoer assignment
Select the morphosyntactic properties of
the argument
Verb selection and voice
Display the verb selection and voice
3
5
6
SEMANTICS-TOSYNTAX
Retrieve the LS and verb from the lexicon
Assignment of arguments to the LS
arguments
PHASE 1F –
DIALOGUE
MANAGER
Ascertain whether there is any missing
information
Refer to the stored discourse referents – if
any, and process with the earlier Phase 1
steps
Attach the performative information to
the Speech Act Construction
PHASE 1G –
CREATE SAP
topics
Display the LS and verb from the
lexicon
Display the argument values
Display the specific elements that
missing
Display appropriately any saved
discourse referents
Display the performative type
information
The analysis of utterance/responses will be constrained to assertive and interrogative (WH-word) SAs. SAC
example shown in Table 2 include: - ‘assertive: ATE’, taken from a sample collection of 34.
The selected SAC, is updated with all the grammatical information.
These include: voice opposition;
macrorole; pragmatic; semantic; lexical; the PSA (privileged syntactic argument)- similar to the ‘subject’ of SV-O; matching signature; syntactic morphology (rules); focus; semantic information such as tense and aspect.
Here the updated SAC will use the generalised lexicon and computationally work with the surface syntax to the
underlying semantic forms. Any discourse referents that are generated are also checked and updated accordingly.
The updated SAC will store each text and the associated complete logical structure (LS) based on the working
semantic predicating element. The linking system will facilitate the syntactic parse to enable fixed word order
for English (SVO), and to unpack the agreement features between elements of the sentence into a semantic
representation (the logical structure) and a representation of the Layered Structure of the Clause (LSC). The
linking system will facilitate procedures for semantic-to-syntax and syntax-to-semantics, parsing and in the
process of formulating a grammatical correct response.
3.2
Design framework - Agent Cognitive Model (Phase 2)
Cohen and Levesque (1988) take the viewpoint of language as action, and view utterances as events (updated to
speech act performatives (SAP) messages) that change the state of the world, and hence speakers and hearer’s
mental state change as a result of these utterances. Further it is necessary for a computer program to recognise
the illocutionary act (IA) of a SA, for both the speaker (USER) utterance and hearer (AGENT) and a response.
The single agent environment will constitute a language model, mental model to work the BDI states, working
model of memory (to reason with states based on current knowledge), knowledge model (world knowledge –
made up of shared and individual beliefs) and dialogue model (to provide a response back to the user). The
next question is how will this integrate and function; this is achieved in the Phase 2 – Agent Cognitive Model
(ACM) with an input of the Phase 1 RRG Language Model. The ACM contains two inner models namely the
Agent Knowledge Model (AKM) and Agent Dialogue Model (ADM). The ACM Phase 2 model has a series of
pre-agent steps and main agent steps illustrated in Figure 5.
The behaviour of agent is identified by the following basic loop where agent iterates the following two steps at
regular intervals: (1) Read the current message, and update the mental state including the beliefs and
commitments; (2) Execute the commitments for the current time, possibly resulting in a further belief change.
The preliminary steps (Panesar, 2017) involve Step Pre-Agent (1) Create Belief Base - the agent’s knowledge;
Step Pre-Agent (2) Map to Message Format - the appropriate Speech Act Performative (SAP) (from the Phase 1RRG Language Model) is re-mapped into a data structure to reduce redundancy and enable efficiency in the
planning and reasoning process.
The main steps (Panesar, 2017) are identified in (1):
(1)
4
a) Step (1) Agent Perceive - to perceive the environment, and ascertain the state of the environment, this
event is added to the event queue (EQ);
b) Step (2) Agent Query - the current beliefs – here the user’s beliefs are queried against the belief base this event is added to the EQ;
c) Step (3) Agent Select Plan – invokes the list of possible plans, and asserts the best way forward; this
event is added to the EQ;
d) Step (4) Agent -Update Belief – if appropriate the agent’s knowledge is updated;
e) Step (5) Agent Select goal – here a goal is selected from the intention stack – stored as part of the plan
library . This could be simple goal such as “True assertion”; or “missing information” for instance, the
location, or the object, or it temporal information;
f) Step (6) Agent Execute Action–involves invoking the Agent Dialogue Model, and the RRG Language
Model for an effective Speech Act performative message back to the Agent;
g) Step (7) Agent Event Handler – handles each different type of event – from the perception (of messages)
, querying (belief base), updating (belief base), selecting (a plan, and further goal), to executing an
action. Prior to checking the BDI states of the user and agent, the SAC is updated to include the
performative information of the message in an agent environment.
Figure 3 - Design framework - CSA Agent Cognitive Model (Panesar, 2017)
Bratman (1987) identifies the workings of how people make decisions and take action in the form of the mental
attitudes of beliefs, desires and intentions - goals (BDI) and in relation to rational interaction. In relation to the
conversation, the methods used will work based on a sentence pair. The user instigates the utterance into the
CSA framework, and it is the agent which decides on the appropriate dialogue and grammatical response. The
sentence pair and sub dialogue internal representations will be stored – based on theories and models such as the
discourse representation theory (DRT) (Kamp et al., 2011). To facilitate conversation, the DM is invoked, and
discourse referents in the previous utterance of the sub-dialogue are resolved. This will serve two purposes: 1)
to establish the NLU of the utterance; (2) to forward to the dialogue model to ascertain a response. This
response generator here will further make use of the language model, and speech act construction model, to
formulate a grammatically correct response.
5
3
Implementation details
Figures 1 to 3 are mapped to an operational framework, and implemented as a Java prototype, developed in
Eclipse IDE platform, predominantly as POJO (plain old java objects) with some API support, based on a food
and cooking domain (ontology) due to its rich and representative (different lexical entries) nature. This
prototype constitutes three phase models: (1) a linguistic model based on a functional linguistic theory – Role
and Reference Grammar (RRG) (Van Valin Jr, 2005); (2) Agent Cognitive Model with two inner models: (a)
knowledge representation model employing conceptual graphs serialised to Resource Description Framework
(RDF); (b) a planning model underpinned by BDI concepts (Wooldridge, 2013) and intentionality (Searle,
1983) and rational interaction (Cohen and Levesque, 1990); and (3) a dialogue model employing common
ground (Stalnaker, 2002). In the next section, we present an evaluation framework for LING-CSA.
4
Discussion of the selection evaluation methodology and testing framework
The purpose of evaluation is to assess the system to see if does what it is supposed to do. Radziwill and Benton
(2017) present a case of reviewing articles on evaluating CSA using terms such as ‘quality assurance’, ‘
evaluation’, and ‘assessment ‘ over the years 2016 and 2017 and concluded that the quality of CSAs was
aligned with the ISO 9241 concept of usability: “The effectiveness, efficiency and satisfaction with which
specified users achieve specified goals in particular environments” (Abran et al., 2003). These outcomes: (1)
efficiency considered performance; (2) effectiveness considered functionality and humanity; (3) satisfaction
considered affect, ethics and behaviour and accessibility. One interesting attribute was in question: whether or
not a CSA should pass the Turing Test (Turing, 1950). According to most researchers, and following the earliest
conversational interfaces, responding and interacting like a human should be the top priority, and subsequently
this principle guided development for nearly four decades since ELIZA (Weizenbaum, 1966) came online.
However, it is critical that the goal of CSA should not aim to act human - is shared by some analysts like Wilson
et al. (2017), and further every implementation will have different goals and will prioritize different quality
attributes at different phases of the system’s lifecycle, using various quality assessment approaches from test
scripts, question-and-answer, linguistic quality, and metric-based to name a few (Radziwill and Benton, 2017).
Subsequently, a customised multi-approach assessment will be adopted to address the unique goals of the LINGCSA based on the phase models, checking performance both at internal processing, interfacing of the models and
their stages and post-implementation assessments, invoking the tracking log as a qualitative analysis. Testing
strategy is driven by: (a) grammatical testing (English language utterances) and NLP tasks; (b) software
engineering (UML modelling, architecture centric, data structures, algorithms and patterns); (c) knowledge
representation (KR) logics (first order logics and graph theory) and important performance indicators for KRR
as: i) representational adequacy ii) inferential adequacy – is the ability to represent the appropriate inference
procedures; (d) agent practice (message passing, planning and cognitive responses and conforming to its
appropriate inputs, environment, actions, performance measures (IEAP) (Russell and Norvig, 2016); (e) A range
of RRG tests applied at various representations to address the goals of RRG - explanatory, descriptive, cognitive
and computational adequacies (Panesar, 2017). A set of evaluation criteria are identified in (3):
(3)
a) Criteria 1- Could the system present a mapping of the syntactic representation to a semantic
representation, for the utterance taking the form of a simple sentence?
b) Criteria 2 - Could the system present an adequate explanation of the NLU of the utterance?
c) Criteria 3 - Could the system demonstrate the SAC use in the manipulation of the utterance?
d) Criteria 4 - Can the dialogue manager interface the language model?
e) Criteria 5 - Could the system demonstrate the agent BDI and knowledge representation?
f) Criteria 6 - Could the system represent the user’s BDI states?
g) Criteria 7 – Could the system query the knowledge base for a fact (from the speech act performative)
h) Criteria 8 - Could the system devise an appropriate plan based on the BDI states?
i) Criteria 9 - Could the system generate a grammatically correct response in RRG based on the agent’s
knowledge?
6
5
Detailed evaluation of the CSA framework
This section evaluates the LING-CSA framework and three phase models with criteria considered by grouping
per model, and assessed with some supporting tests, and their summative findings and screen captures.
5.1
RRG model evaluations
RRG is a linguistic theory invoked to facilitate the implementation of a linguistic engine for the NLP and
understanding for the CSA. RRG concepts will be used for the linguistic manipulation of the utterance and the
response. Criteria (1-4) are reiterated below:
5.1.1
Criteria 1- Could the system present a mapping of the syntactic representation to a semantic
representation, for the utterance taking the form of a simple sentence?
Criteria 2 - Could the system present an adequate explanation of the NLU of the utterance?
Criteria 3 - Could the system demonstrate the SAC use in the manipulation of the utterance?
Criteria 4 - Can the dialogue manager interface the language model?
Criteria 1 – Syntactic and semantic representation
The syntactic representation (layered structure of the clause - LSC) and the semantic representation (logical
structure LS) are part of the basic concepts of the RRG functional linguistic theory. Further in the CSA, the LS
and LSC work hand in hand together via the RRG linking system, and the semantic-to-syntax algorithm, and the
syntax-to-semantics for the production and the comprehension of the utterance. As part of the implementation
the linguistic algorithms and principles have been consulted, and developed into CSA logics. Table 3 identifies
a range of utterances used for checking a snapshot of syntactic and semantic representations.
The utterance example ‘they are hungry’ is highlighted in Figures 4 and 5. Here the dialogue management work
is resolved. The tense, aspect, privileged syntactic argument (PSA), actor-undergoer hierarchy (AUH) and case
marking are also assigned, and LSC and LS representations generated.
Table 3 - Set of utterances for testing RRG linking system
No
Utterance
1
Gareth ate
2
Gareth eats the
pizza
Gareth is
hungry
3
4
5
Gareth ate the
pizza quickly
Yesterday
Gareth ate in the
pizza
6
The pizza is in
the oven
7
Where is the
pizza
8
Gareth ate the
pizza in the
restaurant
Type
of
verb
Intransitive
Verb ‘ate’
Verbal noun
Copula verb
with an
adjective
Transitive
Transitive
Copula verb
and
be-LOC
WH- word
Copula verb
Transitive
Logical Structure info and number of nominals in the LS
Syntactic representation (SR); Expected results
1 argument and 1 nominal
LS (do (ate) and main clause (SR)
2 arguments and 2 nominals
LS (do (eat) and main clause (SR)
1 argument and 1 predicating argument
LS (be) and Main clause (SR)
Actual
results
√
2 arguments
LS (do (eat). SR has a main clause and ‘quickly’ is a periphery
2 arguments
LS (do (eat)
SR has a left detached position (LDP) yesterday’ and main clause
has ‘quickly’ as the periphery constituent.
2 arguments
LS (be (is) and main clause has ‘the pizza’ and ‘the oven’ are both
noun phrases (NP).
1 argument, 2 nominals.
Here ‘where’ is replaced in the first argument position as a nominal
of the LS.
SR – ‘where’ token takes on noun phrase (NP)
3 arguments and 2 LS
LS (do (eat) and be-in and main clause (SR)
Main clause and ‘in the restaurant’ is a periphery preposition (PP).
√
√
√
√
√
√
Partially
7
Figure 4 - Utterance - 'they are hungry’ - LS and case marking
Figure 5 - Utterance - ‘they are hungry' - LSC and LS representations
Table 3 highlights a range of snapshot utterances, from the full range of the tense and aspect spectrum. Here the
auxiliary verb is followed by a verb or an adjective, and every utterance has a matching speech act construction
and their respective signature that can be processed by the CSA.
Example 9 Analysis - ‘Gareth ate the pizza in the restaurant’, it worked partially.
The syntactic representation was generated correctly as:
SENTENCE ( CLAUSE ( <CORE> <NP> gareth ( <NUC> ( <PRED> <V> ate ) ) ( <NP> ( the
pizza ) ) ( PERIPHERY ( in ( <NP> ( the restaurant ) ) ) )
However the semantic representation is:
<IF>ASS<TNS>PT do’(gareth,[eat’(gareth,restaurant)]) & INGR
consumed’(restaurant)<IF>ASS<TNS>PT be-in’(gareth,restaurant)
8
Here the propositional assignment rules have been applied, but some refinement of logic rules is still necessary,
to ensure the macrorole assignment is accurate, using a range of agreement features. From the tense and aspect
list a few entries have been challenging for the LS. For example, Gareth was invited to dinner, presents two LSs.
Alternatively, ‘she has to eat dinner’ has two LSs with an outputLS as:
<IF>ASS<TNS><PST>
do’(she,[eat’(she,dinner)]) & BECOME consumed’(dinner)<IF>ASS<TNS><PST> has'(she,dinner).
From a LSC perspective, some of the syntactic representations include the auxiliary verb and not the main verb.
For example, ‘I can cook dinner’ creates a correct semantic representation, but a syntactic representation with
one token missing.
To summarise for this criteria, the CSA can produce both a semantic and syntactic representation of an utterance,
in most cases accurately. The specific limitations are discussed in the final section.
5.1.2
Criteria 2 – Natural language understanding
For the pre-analysis tasks – tokens were checked for viability, and furthered stemmed. The WordNet API was
tested for some stems – which provided inappropriate answers for this model – hence a stemming algorithm was
coded. For example in the CSA, the token ‘serving’ would be reduced to the base verb of ‘serve’ as opposed to
‘serv’ an answer provided by the WordNet API. The base verb of ‘serve’ is ‘serve’, and the correct SAC is
selected, with a range of valid signatures. Figure 6 identifies the base verb.
Figure 6 - Stemming feature to extract the base verb
5.1.3
Criteria 3 - Speech Act Constructions (SAC)
NLP sentences make use of lexical, syntactic, semantic and pragmatic information in order to fulfil the role of
conveying meaning. The SAC is a grammatical object/container to contain some default information for the
speech act type, and has the capacity for a series of computational updates.
For RRG manipulations, it is the application of speech act theory to utterances and conversations, which
facilitate the interaction with the agent cognitive model and dialogue model. Once an utterance is uttered it is the
illocutionary act – what is to be done – which takes precedence. Here the illocutionary act could be an assertive
or interrogative statement (WH-words), with a desire for the hearer, to form some correlative attitude and hence
act in a certain way. The adoption of SAC has worked well, both from the point of view of the selection of the
SAC, and stages of populating of it, from the application of the RRG linking system. This is demonstrated in
Figure 7.
9
SACs effectively extends to create a speech act performative (SAP) to form a container for a message that will
form an input in to the Agent Cognitive Model, as shown in Figure 8. This SAP with its seventeen attributes is
available in the agent environment to establish the user’s belief states and further support the querying of the
agent’s knowledge base and planning a response back to the user.
To summarise, the SAC as a grammatical object together with the rich RRG Lexicon has enabled an effective reorganisation and integration of the RRG Linking system.
Figure 7 Selection of the Speech Act Construction
Figure 8 - Speech Act Performative – message format > Agent Cognitive Model
4.2.4
Criteria 4 - Dialogue manager
In the LING-CSA, after the standard utterance analysis steps, the dialogue manager facility is an interim process,
which initially automatically checks for missing information or whether reference has been made to the other
discourse elements, such as the pronoun category as demonstrated in Figure 9 Dialogue Preparation (1).
Dialogue Preparations (2) is involves making use of the current utterance and further extracting and saving the
nominals for future utterances. The nominals are saved in an output file ‘DiscourceReferents.txt’ which is saved
locally in the resource folder of the CSA program. The file contains a token with its associated POS tag. This
information will be used by another utterance, via another execution of the CSA program.
10
Figure 9 - Dialogue Preparations (1) and (2)
Taking the example, ‘he is in the restaurant’, Figure 9 illustrates the re-loading of these nominals in microlexicon. The new utterance has a ‘he’ which is a pronoun that needs resolving, and its own discourse lexicon is
generated as well.
Figure 10 - Dialogue management (3)
Through a series of rules and using the agreement features such as gender, person, definiteness, animate and
human, the pronoun can be resolved as demonstrated in Dialogue Preparations (3) and (4) in Figure 10 and
Figure 11 respectively. The new utterance generated is ‘Gareth is in the restaurant’ and thus can go forth for
further RRG processing to the selection of a SAC.
To summarise this dialogue management makes use of simple discourse representation theory (DRT), and forms
a reusable feature as part of the agent planning process to formulate a grammatical correct response. For this
reason the Dialogue Model computationally has been combined with the Agent Cognitive Model.
11
Figure 11 - Dialogue management (4)
4.1.5
Assess the CSA in terms of the goals of the linguistic theory
In this section we have demonstrated that criteria 3 (a-d) have been achieved as part of the language model. In
order to ascertain the success of how RRG has performed as a linguistic engine for the CSA, it is important to
assess the goals of a linguistic theory, and critique them, with the main theoretical characteristics of RRG in
relation to grammatical relations.
Starting with (a) describing linguistic phenomena (central goal) - the LING-CSA can describe the utterance at
various linguistic levels such as the syntax (layered structure of the clause). Next (b) explaining linguistic
phenomena - the CSA provides semantic representations (logical structure) of simple sentences in English,
which provide an explanation of the utterance, as well as the representation of meaning. Next (c) language
processing and knowledge - has been achieved in the CSA, via invoking the RRG lexicon, as a form of
knowledge of the language, and the selection of a SAC to facilitate the producing and the comprehension of the
utterance.
Further, the syntax-to-semantic and semantic-to-syntax algorithms provide a framework for
processing language – for example, take a ‘wh-word’, it is the semantic-to-syntax algorithm which states that the
‘wh-word’ as a nominal will replace the first argument in the first argument position of the LS. Finally, for (d)
computational adequacy – LING- CSA provides a computational framework of the RRG linking system and
embedding an extension of speech act constructions.
To summarise this implemented RRG Language Model is fit for purpose as a linguistic engine for the LINGCSA framework (Panesar, 2017).
12
4.2
Agent Cognitive Model evaluations
LING-CSA’s second phase of the Agent Cognitive Model has two inner models: the Agent Knowledge Model
(AKM) and the Agent Planning Model (APM). The ACM composes the agent environment with a knowledge
base, and takes on an input as a speech act performative (SAP) – a message with a propositional content
reflecting the message send by the user. The elements of the speech act performative (SAP) will be used by
various functions of the agent querying and planning process to generate a response. Figure 12 presents the class
diagram for the ACM, linking with the RRG language model.
Figure 12 - Class Diagram of the Agent Cognitive Model
4.2.1
Knowledge Model evaluations
A standard known as the COGXML (Conceptual Graphs Extensible Markup Language) facilitates the exchange
of graphs, and the connection of different knowledge systems to encode and decode CGs. CGs can be easily
translated to RDFS (Yao and Etzkorn, 2006), and its evolution OWL (Horrocks et al., 2005) applied to the
Semantic Web space. This is achieved via COGUI which is able to export projects to RDF with RDF/XML,
whereby only type hierarchies and facts are exported, but rules, constraints and type disjunctions (banned types)
are ignored (GraphIK, 2012). Rules, banned types and constraints were successfully tested in COGUI.
13
In the first instance, the exported RDF/XML file is independently checked with a W3C RDF Validator, with
success of a range of RDF triples. As the rules, constraints and banned types are ignored the OWL ontology
rules will have to be deployed. In this section, criteria 5 and 6 are reiterated:
Criteria 5 - Could the system demonstrate the agent BDI and knowledge representation?
Criteria 6 - Could the system represent the user’s BDI states?
4.2.2.1 Criteria 5 – Agent BDI and knowledge representation
CGs created in COGUI are serialised into RDF/XML output form and parsed into RDF triples to form the
agent’s knowledge base. Figure 13 and Table 3 demonstrate these two stages.
Figure 13 -COGUI - a set of facts for the food and cooking domain
Table 3 - An extract of the RDF triples -forming the agent's KB
No
1
2
3
4
Subject
http://www.lirmm.fr/cogui#ct_ad452f18e654-4ae6-b3a1-b7320616283b
http://www.lirmm.fr/cogui#ct_fdc6d7d01314-4fb7-8428-51e122953250
http://www.lirmm.fr/cogui#ct_ad452f18e654-4ae6-b3a1-b7320616283b
http://www.lirmm.fr/cogui#ct_ad452f18e654-4ae6-b3a1-b7320616283b
Predicate
http://www.w3.org/1999/02/22-rdfsyntax-ns#type
http://www.w3.org/1999/02/22-rdfsyntax-ns#type
http://www.w3.org/2000/01/rdfschema#subClassOf
http://www.w3.org/2000/01/rdfschema#label
Object
http://www.w3.org/2000/01/rdfschema#Class
http://www.w3.org/2000/01/rdfschema#Class
http://www.lirmm.fr/cogui#ct_fdc6d7
d0-1314-4fb7-8428-51e122953250
"Breakfast"@en
In summary this criteria has been achieved, as RDF triples can be queried for specific facts and rules using a
SPARQL query, based on the subject-predicate-object (S-P-O) concept.
4.2.2.2 Criteria 6 – User’s BDI states
The SAP is a message format to the Agent environment, and once listened and acknowledged this message is
saved. As the SAP has been generated as an output from the RRG Language Model, the data structure and the
algorithms to formulate the original SAC are available for reuse. The SAP has seventeen attributes which are
each saved in their respective data structures in the agent environment. Here the propositional content ‘Gareth
ate the pizza’ and the LS elements are used to establish the user’s BDI states, as identified in Table 4.
14
Table 4 -An example of the user's BDI states
Speech Act Performative attributes
Propositional content
Logical Structure
PSA, actor and undergeor
User’s BDI states
Example
‘Gareth ate the pizza’
<IF>ASS<TNS>PT do’(gareth,[eat’(gareth,pizza)]) & INGR consumed’(pizza)
Gareth, Gareth and pizza
Belief state – Gareth
Desire – ‘eat’
Intention – ‘consume pizza’
4.2.2.3 Assess the CSA’s knowledge representation
CGs was selected for their effective representation of NL text (Sowa, 2009). The serialisation to RDF/XML
format and further parsing to RDF triples was achieved due to their common graph theory. Further the RDF
triples facilitate SPARQL querying (S-P-O) format and hence is a sound choice, and has been achieved. The
computational solution of the user’s BDI states has not been demonstrated, as on one hand the ground work has
been established with the RRG Language Model, particularly the SAC and the LS, and the accessibility and
reuse of their data structures and algorithms. On the other hand, a major technical issue presented an obstacle to
the further use of the user’s BDI states, and querying of NL text against the RDF triples. This issue has had an
impact on the future development of LING-CSA (Panesar, 2017), and explained in the following section.
Morphology involves NL syntax composed typically of sophisticated words, phrases and clauses formed
together to construct meaningful sentences, governed by grammatical rules. Similarly, multiple words combined
can form constructs that can form a subject, predicate or object (direct or indirect). Comparatively, ontology and
domain knowledge have a syntax limited to S-V-O sentences, for example, ‘Mechanic repairs equipment’ is
called a triple, with passive and active statements, but having
no ontological grammar rules to create new
‘multi-word construct’ which will take that form of a class and property. For example, in NL the separate words,
lacrosse, sport, and equipment, can form the phrase ‘lacrosse sports equipment’, comprising constituent words
and their individual meaning. In ontology, this will be conflated to sports as a single canonical form.
Taking a further NL sentence example ‘An action is the subject of the action functional association’ as illustrated
in Figure 7, it is syntactically conflated to SVO triple ‘action’, is-the-subject-of’, ‘action-functional association’.
These syntactic limitations give rise to syntax conflation which provides a serious of impacts such as (1) SVO
triples are void of morphology (SVO do not vary in meaning – for instance tense and number); (2) a complex NL
syntax will be reduced to a set of simple SVO triples; (3) the SVO construct will have one and only one named
element; (4) The lack of function words makes to its difficult to connect different SVO triples to form compound
sentences or form a discourse context.
Figure 14 Complex NL sentence to a conflated ontology statement
15
Here the NL syntax is flattened into three ontology strings, whereby the original NL sentence lost its internal
constituent phrase structure (phrase-splitting whereby the prepositions and predicative nominative) are
completed into the relation name, and the object of the proposition is conflated to a class name.
The original NL sentence is conflated to a simple SVO triple with three ontology words. Linguistically, this
demonstrates a restriction of noun-verb-noun structure with morphology, lexical and syntax limitations adding to
the cascading conflation (Bimson and Hull, 2016) as identified in Figure 14. Languages are semantically more
expressive than ontologies which mean that there is a major challenge, in translating NL semantic to ontology
semantics, leading to a significant meaning loss, and characterised in terms of morphology, lexical and syntactic
– semantic gaps.
The next cause of action is to reduce this semantic gap, by “building a lexical bridge (LB)” (Bimson et al.,
2016) between the NL semantic and ontology semantics, with an aim to capture more of the meaning, by
attempting to ‘lexicalize the ontology’. This will entail, parsing the ontology literals (class and property string
names) to more lexicalised items, thus forming an ontology lexicon (OL). A further step can be taken to check
the semantic equivalence of the ontology lexical meaning with a range of NL semantics. This OL can be used to
identify and eliminate any redundant knowledge, such as RDF assertions, to find any hidden lexical meaning.
5.3
Considerations of a lexical bridge for the Agent Cognitive Model
This lexical bridge (LB) will form a component of the ACM of the LING-CSA specifically as an extension to
the knowledge model. The initial KB is to be expressed in CGs and serialised into RDF/XML and then exported
into RDF triples for further representation and reasoning. The LB will support this representation, to allow
querying and facilitate any reasoning activities. Figure 15 below outlines the LB demonstrating the semantic
crosswalk form the NL semantics to the RDF semantics based on parsing the RDF triples into NL words.
Figure 15 Lexical Bridge for the CSA's belief base + BDI Parser to resolve the agent’s BDI states
This LB will require: (1) RDF parser to parse the RDF Triples into NL words, against the common RRG
Lexicon (or to generate a RDF lexicon). (2) To reuse an existing NL parser (available in RRG Model) to
algorithmically parse NL text into words, parts of speech from Phase 1 – RRG model using the RRG lexicon (3)
an ontology-to-NL mapping-algorithm – comparison of the RDF triples with the NL words.
The current design framework of the Agent Cognitive Model in Figure 4, presented no opportunity to compute
the agent’s BDI states, as they are dependent on the KR of the agent’s belief base, thus we need a sound model
to query the belief base using NL text. Hence our dilemma is linked to the previous issue and resolution, in that
once the LB and mapping algorithm computations are done, the resultant output together with a BDI parser, will
support the deduction of the agent’s BDI states as illustrated in Figure 4. A range of pseudo coding steps for the
LB are presented in Table 5.
16
Table 5 Pseudo coding steps are identified for the Lexical Bridge
Lexical Bridge (LB) Algorithm
1.
Translate the RDF Triples class or property literals into its component lexical items (using RDF Parser) - to determine
the constituent words and POS.
2. Standard parsing of semantic equivalent NL text into lexical items (using the NL Parser invoking the RRG Lexicon)
3. Comparison of the RDF lexical components to NL text to determine their semantic equivalence.
Table 6 RDF object - semantic equivalent NL words in the Lexical Bridge – Step 1
RDF triple object property
Has-Food-Ingredient
Natural Language paraphrase
Food contains MealTypes
Gareth has eaten
Egg is an ingredient of omelette
Continue parsing with all RDF object property literals, to complete the lexicalisation process, invoking the RRG
Lexicon as identified in Table 7.
Table 7 Parse with corresponding natural language texts – Step 2
RDF concept
Has-Food-Ingredient
Food contains mealtypes
Egg is an ingredient of
omelette
Word #
1
2
3
1
2
3
1
2
3
4
5
6
Word
has
food
ingredient
food
contains
mealtype
egg
is
an
ingredient
of
omelette
POS tag
POS
N
N
N
V
N
N
VBE
ART
NOUN
PRP
NOUN
Noun
Noun
Noun
Transitive verb
Noun, plural
Noun
Indefinite article
Noun
Proposition
Noun
Base verb
has
food
ingredients
food
contains
mealtype
egg
is
an
ingredient
of
omelette
Step 3 from Table 5 - requires an algorithm to compare the lexical semantics in the RDF form to the Natural
Language Semantics – focusing on the RDF parse trees, and comparing the RDF words (RW) to text words
(TW) in the text parse trees, as demonstrated in Table 7.. This algorithm will invoke Description Logic (DL)
(objects having features), which is in effect a mapping algorithm of the words using a RDF equivalency
algorithm, which is based on standard graph algorithms (Dijkstra, 1959). This LB could potentially provide
significant benefits to the deployed CSA semantic solutions: (1) RDF redundancy identification and prevention;
(2) improving semantic search in text; (3) Bridging the semantic gap between ontology semantic and NL
semantics (4) semantic comparison analysis of ontologies. Simultaneously, the LB demonstrates aspects of
simplicity as well as inherent complexity to support our semantic gap goal.
Criteria (7), (8) and (9) – Agent Planning Model and Dialogue Model
4.3
The Agent Planning Model (APM) and Agent Dialogue Model (ADM) are interlinked in the Agent Cognitive
Model (ACM), at the implementation level. To generate a response, the Dialogue Model is invoked, which
further invoke the language model. The dialogue model has a dual role in the analysis of the utterance and in the
generation of the response. In this section we address criteria 3(f-h):
Criteria 7 – Could the system query the knowledge base for a fact (from the speech act performative)
Criteria 8 - Could the system devise an appropriate plan based on the BDI states?
Criteria 9 - Could the system generate a grammatically correct response in RRG based on the agent’s
knowledge?
17
The remaining three criteria are examined together due to their dependencies. To start, the ACM
interconnectivity is outlined. Figure 12 illustrates how the RRG Language Model and Knowledge Model
interface with the Agent Cognitive Model. Further it demonstrates the dialogue handling process to support the
generation of a response. Conceptually the Dialogue Model is embedded in the implementation of the ACM and
further the output from the dialogue handling process, will be a response sent back to the RRG Language Model
to refine to a grammatically correct response. Figure 16 presents a class diagram of Java based classes in
Eclipse IDE platform demonstrating our ground work which evidences (a) Workings of the agent environment;
(b) Perceiving input of a speech act performative (SAP) message; (c) Processing of the user’s BDI states; (d)
Querying of the belief base; (e) Querying of user’s BDI states against the agent’s BDI states; (f) Event handling
process; (g) Process for the selection of plans; (h) Process to carry out a plan based on the intention and goals; (i)
Dialogue handling process to work towards a response invoking the construction of the predicate class category
(type of verb – transitive, intransitive, modal) of the response.
Figure 16 - Class Diagram - Agent Cognitive Model - Classes in Eclipse
Criteria (f) – querying of the agent’s KB against the content (for example, ‘who ate the pizza’) from the SAP
message, posed a significant technical challenge for the continued work on the Agent Cognitive Model. To
summarise, the propositional content is in a NL semantic format, and the RDF triples are in an ontological
semantic format, and they do not map together, and present a semantic gap. This technical issue posed a massive
challenge similarly identified by (Bimson and Hull, 2016). This technical issue has been explored and an
abstract conceptual framework of a lexical bridge utilizing the RRG Lexicon with a mapping algorithm has been
proposed to solve this problem, and further facilitate the querying of the KB.
Criteria (g) relating to the representation the agent’s BDI states was also an issue, and was explored by
extending the lexical bridge with a BDI parser.
Criteria (h) will be achieved by adding a next layer of classes to reflect the technical solution of the lexical
bridge, mapping algorithm and the BDI parser. The dialogue model has a dual role in the analysis of the
utterance and in the generation of the response.
In summary, the groundwork for the agent environment and planning model has been demonstrated, with a
working knowledge model, and a proposed conceptual framework of lexical bridge solution.
18
6
Evaluation conclusions and recommendations
To iterate, the goal of this linguistically centred CSA (LING-CSA) was to investigate the accuracy of the
interpretation of meaning to provide a realistic dialogue to support the human-to-computer communication.
LING-CSA was multi-disciplinary invoking natural language processing, computational linguistics, knowledge
representation, functional linguistics, agent thinking and artificial intelligence, as demonstrated in the conceptual
and operational framework of LING-CSA. Subsequently, by reviewing related works, a customised evaluation
methodology and testing framework was selected, driven by linguistic theory goals, phase models, RRG
grammatical structures, the specific speech act at play and agent environment and the functions of a
conversational software agent. A set of evaluation criteria was devised to exhaustively assess where the
requirements of linguistically centred CSA were being met.
6,1
RRG Model evaluations
The RRG phase model evaluations identify that RRG is fit-for-purpose as the linguistic engine for LING-CSA,
and has a range of improvements and recommendations, (Panesar, 2017). For this paper, a subset is presented.
However, one obstacle – Example 1 identified is the workings of two logical structures (LSs) and three
macroroles. This issue leads to problems with irregular verbs and exceptional transitivity (number of macroroles
is greater than two), and further Van Valin Jr (2005) justifies and assumes a third macrorole for two reasons: (1)
that the third argument is that of a ditransitive verb, (2) or to accommodate a dative case assignment. In some
cases, there are some complex LSs, for the verbs give, present, show, teach, load and put. In mapping the
semantic representation, with a non-macrorole assignment, the AUH and ranking is applied but, there is further
dilemma as to how the non-predicating elements will behave. The proposition assignment rules for English deal
with this situation, where there is a ‘…..to…”, ‘….from…’, or ‘with’ is achieved by propositional stranding, and
central to extending the current prototype of the RRG Language Model. Example 2 - To refine the
computational rules for anaphoric resolutions, for example, for other pronouns such as, 'my, your, his/her, ours,
your, their'. The current solution is simple and effective, and can be extended. Example 3 - as part of the
semantic-to-syntax algorithm, the WH-word is correctly substituted for a noun phrase, but further work with the
propositional stranding rules and argument (lexical features) is necessary, as there is links with Example 1.
As noted a snapshot of key recommendations are included: (1) consideration of the requirements for complex
sentences (more than one clause) with the extended use of procedures for the linking system; (2) support for
multi-lingual (additional lexicons) such as the Spanish, co-operating with RRG scholars; (3) consideration of
other SA categories such as commissives and emotives, such as, for the automatic analysis of social media
tweets; (4) to include superlative adjectives and adverbs RRG lexicon to enforce greater coverage.
6.3
Agent Cognitive Model recommendations
The BDI architecture is based on the ‘intentional stance’ which is the highest level of abstraction of human
reasoning where the agent will do what it believes are its goals. There are two main areas for future research,
based on this evaluation. Firstly, is to reduce the gap between knowledge and language, in the agent space.
Here, language is in the context of a RRG linguistic representation against the RDF Triples representing the
agent belief base posed the key obstacle of interoperability of low level mapping and representation. This has
been termed the ‘lexical bridge’ (LB). Central to this LB is the re-use of the RRG lexicon and NL parser. The
derivation of a mapping algorithm from OWL to a NLP representation is recommended, specifically invoking
the RRG lexicon and RDF parser. This mapping algorithm will be interfaced with a further model known as the
BDI Resolution composing a BDI parser motivated by description logics (DL) to derive the BDI states of the
agent. This step will form an extra step to the pre-agent steps of ACM. The algorithm’s output representation,
will further invoke the rest of the ACM steps of querying the knowledge base for a specific belief, creating
planning schemes, selecting a plan, manipulation of the appropriate intention, to create a grammatical correct
response using the RRG linguistic engine and the extended speech act constructions.
A second recommendation is based on the evaluations to the ACM is for an extended design framework to
enable a multi-agent environment where there are both BDI logics and plan-based speech acts to formulate the
dialogue response, and thus will require a more refined evaluation methodology and framework.
19
References
ABRAN, A., KHELIFI, A., SURYN, W. & SEFFAH, A. 2003. Consolidating the ISO usability models. . In proceedings of
11th International Software Quality Management Conference, 2003, pp. 23-25.
BIMSON, K. D. & HULL, R. D. 2016. Unnatural Language Processing: Characterizing the Challenges in Translating
Natural Language Semantics into Ontology Semantics. In: WORKMAN, M. (ed.) Semantic Web: Implications for
Technologies and Business Practices. Cham: Springer International Publishing.
BIMSON, K. D., HULL, R. D. & NIETEN, D. 2016. The Lexical Bridge: A Methodology for Bridging the Semantic Gaps
between a Natural Language and an Ontology. In: WORKMAN, M. (ed.) Semantic Web: Implications for
Technologies and Business Practices. Cham: Springer International Publishing.
BRATMAN, M. 1987. Intention, plans, and practical reason, Cambridge, Harvard University Press.
COHEN, P. R. & LEVESQUE, H. J. 1988. Rational interaction as the basis for communication. DTIC Document.
COHEN, P. R. & LEVESQUE, H. J. 1990. Intention is choice with commitment. Artificial Intelligence, 42, 213-261.
DIJKSTRA, E. W. 1959. A note on two problems in connexion with graphs. Numerische mathematik, 1, 269-271.
FRANCOIS, J. 2014. Functionalist views on the genesis and development of grammar 25th European Systemic Functional
Linguistics Conference Paris.
GRAPHIK. 2012. CoGui (Conceptual Graphs Graphical User Interface) User Guide [Online]. Available:
http://www.lirmm.fr/cogui/newdoc/cogui-index.html [Accessed 12 March 2014].
HORROCKS, I., PATEL-SCHNEIDER, P. F., BECHHOFER, S. & TSARKOV, D. 2005. OWL rules: A proposal and
prototype implementation. Web Semantics: Science, Services and Agents on the World Wide Web, 3, 23-40.
KAMP, H., VAN GENABITH, J. & REYLE, U. 2011. Discourse representation theory. Handbook of philosophical logic.
Springer.
LESTER, J., BRANTING, K. & MOTT, B. 2004. Conversational agents. The Practical Handbook of Internet Computing.
MAO, X., SANSONNET, J.-P. & LI, L. Textual Conversation Agent for Enhancing Attraction in E-Learning. Proceedings
of International Conference on Computer Science and Information Technology, 2012. 49.
NOLAN, B. 2014. Constructions as grammatical objects : A case study of prepositional ditransitive construction in Modern
Irish. In: NOLAN, B. & DIEDRICHSEN, E. (eds.) Linking Constructions into Functional Linguistics: The role of
constructions in grammar. Amsterdam/Philadelphia: John Benjamins Publishing Company.
PANESAR, K. 2017. A linguistically centred text-based conversational software agent. Unpublished PhD thesis, Leeds
Beckett University.
PEREZ-MARIN, D., PASCUAL-NIETO, I. & GLOBAL, I. 2011. Conversational Agents and Natural Language
Interaction: Techniques and Effective Practices, Information Science Reference.
RADZIWILL, N. M. & BENTON, M. C. 2017. Evaluating Quality of Chatbots and Intelligent Conversational Agents.
arXiv preprint arXiv:1704.04579.
RAO, A. S. & GEORGEFF, M. P. BDI Agents - From Theory to Practice. ICMAS, 1995. 312-319.
RUSSELL, S. & NORVIG, P. 2016. Artifical Intelligence: A Modern Approach - Global Edition, Pearson.
SEARLE, J. R. 1969. Speech acts: An essay in the philosophy of language, Cambridge, CUP.
SEARLE, J. R. 1983. Intentionality: An essay in the philosophy of mind, CUP.
STALNAKER, R. 2002. Common ground. Linguistics and philosophy, 25, 701-721.
TURING, A. M. 1950. Computing machinery and intelligence. Mind, 433-460.
VAN VALIN JR, R. D. 2005. Exploring the syntax-semantics interface, CUP.
WEIZENBAUM, J. 1966. ELIZA—a computer program for the study of natural language communication between man and
machine. Communications of the ACM, 9, 36-45.
WOOLDRIDGE, M. 2013. Intelligent Agents. In: WEISS, G. (ed.) Multiagent Systems. 2nd edn. ed. USA: Massachusetts
Institute of Technology.
YAO, H. & ETZKORN, L. 2006. Automated conversion between different knowledge representation formats. KnowledgeBased Systems, 19, 404-412.
20