Remembering a Conversation – A Conversational Memory Architecture for Embodied Conversational Agents

1.1 The Temporal and Functional Aspects of Memory

Memory has two aspects of interest to us here – the temporal and the functional. From the temporal standpoint, memory can be long-term or short-term. Short-term memory often takes the form of working or belief memory that serves to temporarily store information/knowledge relevant to the current situation. It has a relatively short life (minutes, up to a few days). Long-term memory represents (semi-)permanent memory.

From a functional perspective, humans are said to have three forms of memory: procedural, declarative/semantic, and episodic. These can be shortly summarized as follows: (i) procedural – stores the knowledge necessary to perform tasks, e.g. how to solve a math problem, how to drive a car; (ii) declarative/semantic – consists of information about the world, e.g. “Columbus discovered America in 1492”; (iii) episodic – sometimes referred to as autobiographical memory – episodic memory maintains a record of the agent’s experiences as perceived by that agent. Autobiographic memory is often used interchangeably with episodic memory in the literature. However, Conway [10] specifically distinguishes the two, stating that episodic and autobiographical memory form two separate memory systems. Episodic memory is contextually dependent and maintains knowledge about specific events [39, 54], for example, “I saw the Mona Lisa when I visited the Louvre last year.” We focus on short-term episodic memory in this work, although we define short term as lasting for days.

1.2 Our Objective and General Approach

Our objective is to create a conversational memory architecture for ECAs that can permit the ECA to remember important aspects (the gist) of a conversation and retrieve them for use when contextually appropriate. Our episodic memory architecture tackles recollection across one or more episode instances. There are three issues related to this task: (i) knowing what to remember, (ii) how to store it efficiently, and (iii) knowing what to retrieve in the context of a conversation and when to do so appropriately. Our architecture uses a combination of existing openly available software and our own special-purpose software to create an effective memory system for conversational memory in ECAs. We focus on the technical aspects of building an episodic memory system for ECAs rather than on the relevant cognitive issues.

It is commonly accepted that a memory system – whether human or electronic – cannot keep in memory all events experienced in the exact form they were experienced. This would overwhelm our storage capacity as well as our ability to retrieve the memories. Therefore, we subconsciously select what information to keep and what to discard. Sun [49] asserts that the more extraordinary an experience is, the more likely we are to retain it permanently in memory. The emotional intensity of an experience also affects the likelihood of remembering it. This is why we all remember where we were when we heard traumatic news. The same goes for conversations. We do not remember conversations verbatim, but rather extract the important parts of a conversation and remember those. In effect, we get the gist of a conversation as well as important factoids about it (time of appointment, names of new contacts, etc.). The basis of our approach is to identify the important elements of a conversation, store them, and index them according to the context in which it was experienced. This context is remembered, and when it resurfaces (later in the same conversation or in a different future conversation), the memory is recalled and questions related to it can be answered.

We define context as highly tiered. That is, the overall context of a conversation is one thing (e.g. the topic, the conversant(s), the location), but we also recognize the presence of microcontexts throughout a conversation. For example, the overall context of a conversation might be between that between two colleagues discussing a particular contract, while a microcontext within the same conversation could be the client’s love for golf and that he should take her for a round of golf soon.

1.3 Assumptions

We assume that all communications between the human and the ECA are verbal, rather than through visual observation or non-verbal sounds. Furthermore, we neglect variations in the tone of the words that may connote a specific affective state by the human (e.g. anger, frustration, supplication, etc.). While these are undeniably important aspects of conversational transactions, we leave them for future research. Furthermore, we leave for others the investigation of memory in such ECAs when applied to serve as long-term companions (companion agents), empathic agents, and game-based action agents. Moreover, we do not address forgetting, as alas, it is much too often a problem in human-to-human business conversations. Finally, although inspired by existing memory models, we do not claim that our memory architecture necessarily reflects or works in a manner akin to human memory. For these reasons, we refer to our work as an architecture and not as a model.

In Section 2, we briefly examine episodic memory models and architectures reported in the literature. In Section 3, we define our unified episodic memory architecture for ECAs that will enhance the naturalness and effectiveness of a conversation between human and ECA. Section 4 describes our experiments, while Section 5 concludes and summarizes.

2.1 Conversational Memory Models

Sun [49] discussed the CLARION model of memory, which includes episodic memory. Sun makes a case that there are three distinctions between types of memory – (i) implicit vs. explicit memory, (ii) action-centered vs. non-action-centered memory, and (iii) episodic vs. semantic. Lim [29] provides a review of the literature on memory models for companion agents. Lim et al. [30] proposed a means to retrieve autobiographical memories with what they call “spreading association,” all in the context of companion agents. However, no evaluation is provided. Campos and Paiva [8] presented their memory architecture called May, which was integrated into an agent. The main thrust of their work was for the agent to develop a sense of “… intimacy and companionship” with its human user. Brom et al. [7] proposed an episodic memory model for what they refer to as intelligent virtual agents that live and interact with a virtual world and must move about this world. However, they did not address conversational memory. Brom and Lukavsky [6] focused on defining what they call full episodic memory, and asked whether full episodic memory is truly necessary. They did not answer the question but proposed some interesting ideas on the subject. Faltersack et al. [13] proposed an episodic memory model (Ziggurat) as a step toward a general episodic memory model. Ziggurat made use of context for memory retrieval but the application is for action-based games, not conversation. Wang et al. [56] reported their use of neural networks to learn traces (episodes) from sensory information and environmental feedback. Their model (EM-ART) included forgetting. Their application was strictly for game agents, not conversation. Ho et al. [20] described a symbolic autobiographic memory model for use in mission-based environments. An interesting aspect of their work is that their agent was able to tell a story to other agents about its experiences stored in its autobiographical memory. Kim et al. [23] reported a memory system that recognizes and saves elements of the conversation to memory for later use in the same conversation. However, their memory model appears to be declarative/semantic and not episodic.

2.2 Memory in Cognitive Architectures

SOAR [26] and ACT-R [1] are two cognitive architectures commonly used today. Their respective creators correlate the use of episodic memory to beneficial behavior in agents [39]. SOAR and ACT-R both advocate the distinction between episodic and semantic memories [39, 48], as originally suggested by Tulving [53, 54], who opined that non-trivial tasks require aspects of both semantic and episodic memories. SOAR and ACT-R incorporate elements of episodic memory into existing declarative memory [17, 39, 48]. In effect, both efforts aim to store autobiographical data for factual declarative knowledge. SOAR and ACT-R operate in a generally similar fashion, although with some important differences whose discussion is tangential to our objectives.

2.3 Memory in Chatbots and ECAs

Kopp et al. [24] presented a multimodal ECA (Max) capable of utilizing short- and long-term memory structures. Max serves as a museum guide that interacts with users in an enjoyable and informative manner, and aims to engage them in mixed-initiative conversations [28].

Bernsen and Dybkjær [4] described HCA, a multimodal conversational agent that interacts with users to discuss Hans Christian Andersen’s works. While all three types of memory functions can be observed in HCA, its episodic memory influenced the conversational agenda by retaining records of the interactions with a given user. It affects HCA’s ability to maintain prolonged dialogs by keeping track of knowledge visited, and preventing repetitive statements [4].

2.4 Memory in Question Answering Characters

Traum and Rickel [51] showed that Steve-based agents are capable of answering questions about the state of their environment. Steve-based agents employ episodic memory as a record-keeping structure. Three virtual question-and-answer (Q&A) agents were presented by Leuski et al. [27] (Sergeant Blackwell), Artstein et al. [2] (Sergeant Star) and by Traum et al. [52] (Hassan). All three characters implemented derivations of a similar architecture. These Q&A characters take the form of virtual humans inhabiting virtual worlds and interact with users through some model of dialog management.

2.5 Comparison to Existing Works

The episodic memory architecture used in our research strives to overcome constraints evident in the systems described above. While we do not presume to account for all the requirements for episodic memory architectures in general, our work aims to further develop conversational agents with realistic human-like behavior.

By providing an analytic review of previous research efforts in ECAs above, we demonstrated that some common features exist across several episodic memory implementations. Namely, we can reuse the concepts of a centralized memory, design for decentralized systems, conversation log feature tracking, and domain modeling. Additionally, our architecture attempts to emulate the characteristics of concurrent and decentralized modules advocated by development tools such as Microsoft’s Robotics Developer Studio (RDS) [34].

Unlike Max, Sgt. Blackwell, and HCA, our episodic memory tackles recollection across one or more episode instances. Prior works [24, 51] attempt to establish coherence in a single episode of interaction with users. We argue that a more realistic collaboration between humans and conversational agents can be established by extending the episodic memory through various encounters with the user. To accomplish this, our memory architecture relies on a greater number of interactions with these users. Furthermore, we designed the episodic memory structures to store several types of information and record more details about the interaction than has been accomplished previously. Hence, in regards to storage, it can be considered more generalizable while being more expressive.

3.1 System Architecture: The Pipeline Approach

Figure 1 illustrates a stack diagram of the four core components in the episodic memory model as implemented in our architecture. These four modules consist of the memory interfaces, a back-end database, a process of contextualization, and some analysis services. The general purpose of each of these components is described in the next few paragraphs.

Figure 1:

Episodic Memory Architecture Stack with Four Core Components.

The memory interfaces are found at the topmost layer of the stack; however, the architecture responds to requests for recalling events that have been contextualized and stored in the database. Our implementation of memory interfaces is in the form of loosely coupled services, which allow communication to occur between the episodic memory and ECAs that use it. A back-end database running on a server is the storage medium for episodes and the internal processes that manipulate conversational information.

The contextualization process is responsible for managing the input of interaction data, storing them in the episodic memory, indexing the episodes, and deciding which episodes are relevant for retrieval as a result of a the conversation. The analysis services are used by the contextualization process to extract information from the dialog. They are inherently extensible in that several custom services can be concurrently deployed to analyze data.

3.2 Capturing the Conversation

Creating conversational memory begins with capturing information from conversations between the ECA and a human. In a multiparty dialog system, users may interact with several client-side tools. A user may interact with our episodic memory in any of two forms, as shown in the sequence diagram of Figure 2. First, he/she may provide sensory input in the form of speech, which will be processed in memory. Alternatively, he/she may request information on previously stored episodes. In either case, access to memory functions occurs through the memory interfaces.

Figure 2:

Acquiring Conversational Data.

In the first of the interaction scenarios, a client side utility acquires raw data from the users – we use speech here. Our episodic memory architecture creates conversational episodes based on transcripts obtained from a speech recognition engine. The contents of these transcripts, as well as the audio data, are uploaded to the back-end database through memory interfaces. We created a simple, graphical use interface (GUI)-based client program that uses Microsoft’s SAPI 5 speech recognition engine to transcribe spoken data. Our interface accepts transcribed text, information identifying the initiating party, binary data associated with the event, and a unique conversation identifier.

Figure 2 depicts two timelines of events that originate from the GUI and the memory interfaces. The processes highlighted, beginning with “Receiving Audio/Video,” pertain to the acquisition of conversational data. By way of example, we can describe the flow using spoken interactions. Events generated by the speech recognition engine are collected into transcripts and uploaded to a centralized memory database system. This system preprocesses these transcripts to look for topics related to the conversation with tools available on the Internet as well as in our algorithm. The resulting information is parsed into a contextual structure accessed at query time and used to present relevant topical information to the user. Topics are deemed relevant based on the frequency with which they appear in conversations.

We next present the process by which our system selects the details of an episode that will be stored into memory. We expand on this database and the contextualization process in the next section.

3.3 Filtering and Storing Conversational Memories

As memory interfaces collect data from spoken conversations, they, in effect, serve as short-term memory buffers for the audio data and transcripts. The contents of these buffers are transmitted to the back-end database server for processing in episodic memory. Once the server receives the data, it performs three primary functions: (i) stores the raw data, (ii) filters and stores information generated through analysis of the episode, and (iii) indexes the raw and generated data. Storing raw data occurs in a straightforward fashion by using a conversation log similar to that used by Bernsen and Dybkjær [4], Traum and Rickel [51], and Traum et al. [52]. The utterances in a conversation are recorded in a table that serves as a conversation log as they occur.

A second function provided by the back-end server for managing episodic memories concerns the generation, filtering, and storage of episodic information. The information generated by the analysis services identifies key information from the transcripts. We discuss the features selected when describing each of the analysis services. Of importance to our current discussion, however, is the identification of topical information in the form of key phrases.

Two of the services used process the conversational utterances by querying online services that have been trained with large amounts of textual corpus data with which to identify named entities and categorize text. Named entities consist of textual elements that can be tagged as belonging to an agreed-upon taxonomy of categories such as person, location, monetary value, etc. We use Yahoo! [57] and OpenCalais [41] application program interfaces (APIs) to perform the named entity recognition. Because these services have been developed by others and have been widely described, we focus instead on a third service that we developed, which identifies key phrases in addition to the entities recognized by Yahoo! and OpenCalais. While these online services can process large volumes of information and extract topics and terms as recognized by their usage in broader news corpora, our algorithm is tailored to capture snippets from short utterances. It captures short, relevant key phrases from a conversation and processes utterances. It is implemented in five stages:

1. Service input stage – this service obtains the conversational utterance data from the contextualization process.

2. Part-of-speech (POS) tagging – we use a Maximum Entropy POS tagger to extract the part of speech for every word in the utterance.

3. Windowed phrase parsing – phrase groups, consisting of up to seven words, are parsed by using OpenNLP’s Treebank parser [42]. Natural language processing (NLP) parsing assigns a structural description to a sequence of words. In our case, the structure is a tree representation of a phrase’s syntax.

4. Thresholding of complex clauses – we calculate the depth of each clause in the parsed tree and extract the deepest ones.

5. Return complex clauses – complex clauses are returned to the contextualization process. This process concludes the key phrase extraction by storing phrases in a table of candidate topics.

The five steps outlined above correspond to the function of certain methods in our system. Please refer to Ref. [12] for details on these methods, including their pseudo code. An example of getting phrases from a complex phrase follows:

• Phrase: I’d like to know more about the renewals part of the program.

• Groups: (i) I’d like to know more about the (ii) the renewals part of the program.

As can be observed from the above phrase, two groups are created from the original phrase. The first window groups the seven leftmost words using a window size of seven. A second window creates a group from the remaining words in the sentences using a window size of six. For each resulting group of words, the method extracts candidate key phrases. This method returns a list of identified key phrases (to be explained shortly). Finally, another method analyzes the collection of key phrases of the original utterance for noun and verb groups and stores these in the XML object to be returned. Another method is tasked with extracting candidate key phrases from windowed word groups. NLP methods are used to preprocess the word groups. By parsing the word groups and finding co-references, a tree structure of the syntax that constitutes the word group is created.

We now describe a case study that will help clarify the implementation and usage of our algorithm, as a prelude to the functional description of the architecture. To further focus this discussion, we provide a sample exchange from a conversation in Table 1. This segment is an extract from a larger episode in which the system acts as the episodic memory for an ECA that converses with a human user and a moderator via synchronous turn taking. By running our algorithm over each of the utterances from the participants, we expect to extract items of relevance to the overall episode. We can deduce from Table 1 that the user shows interest in learning about renewals in the institution’s program that the ECA references at the beginning of the conversation. Hence, we also expect the list of key phrases extracted to include items with the word “renewal”.

To accomplish the extraction of terms from Table 1, we adapt certain NLP techniques to unstructured, shared-initiative conversations as follows. First, we perform POS tagging over each utterance received. In structured text, tagging might occur at the sentence level. However, speech recognition engines do not provide grammatical punctuation. Instead, utterances appear to behave in a stream-of-thought fashion similar to long sentences. Maximum-Entropy POS taggers, such as that developed by the OpenNLP group [42], which estimate probability distributions given a set of constraints, may perform erratically over longer transcript sentences because of the increased likelihood of a recognition error. Liu et al. [31] demonstrate that Maximum Entropy classifier performance degrades with increased errors in speech recognition output, which is likely to happen over longer and more complex speech. In view of this, we adopt a windowing technique through which we tag and analyze user utterances over five- to seven-word window segments.

As a second step, our term extraction algorithm performs phrase parsing over the windows using OpenNLP’s statistical parser. The reader should note that during parsing, the sentence or phrase is analyzed to determine the constituent groupings of the sentence (i.e. noun groups, verb groups, etc.). These groupings, in turn, may be represented in a tree-like structure.

During the third step, our system creates a structure representative of the tree. From the example of Table 1, the tree results from processing the user’s request for “renewal” information. The target key phrases of the sentence that we wish to extract are found in the tree at a position where the complexity increases. The complexity of a clause in a sentence is determined by the depth in the tree of the grouping in which it is contained.

We now proceed to the final stage of our algorithm. In this stage, the parser’s output is used to determine the depths of all nodes within the tree. Nodes with a complexity level above a threshold are retained. In other words, we place a lower limit on the structural complexity level of the clauses that are considered for term extraction. We expect that the complexity of the clauses will support the notion that terms extracted are items of significant relevance to the conversation. Our approach at extracting relevant terms can be compared to the more complex fact extraction algorithms presented by Pasca et al. [44]. While we exploit both patterns, Pasca et al. only use the POS tags to generalize patterns for fact extraction, and generate a low-quality set of candidate facts. In our implementation, however, we analyze patterns and complexity at the clause level and do not generalize patterns because the dataset available per conversation is limited.

The analysis service that runs our algorithm generates a set of key phrases for each utterance that are returned to the contextualization process. As part of this process, the resulting phrases are stored in a table of candidate topics. The table maintains an index for the records based on a conversation identifier, speaker, and time to facilitate future retrieval.

A summary listing of the topics occurring in an episode can be obtained by filtering the table that stores the candidate topics. Filtering occurs by processing the topics in two steps as follows:

1. Linguistic filter – first, a filter is applied on the table to return key phrases that contain one or more noun or verb groups, excluding prepositional phrases, single adjectives, or single adverbs.

2. Ranking – these groups are then ranked by the number of times they occur in a conversation. While this ranking method uses only a simple frequency heuristic, using a linguistic filter, such as our clause extraction algorithm, in combination with frequency ranking has been shown effective in similar studies [14, 15, 21].

It may be possible to produce a ranking by using more complex measures of interrelationships between extracted key phrases. For instance, Pointwise Mutual Information (PMI) [9] provides a well-known measure for determining word collocations. Word collocation refers to sequences of words, such as key phrases, with group-based syntactic and semantic characteristics and the definition of which results from the grouping and not the individual meaning of the words. Additionally, PMI may be used as a mechanism to gauge how much one word or group of words is related to another. However, PMI may perform badly in a sparse environment for which relatively few words or word collocations are available and shows bias toward infrequent terms [43]. Such a scenario can occur, for instance, in short conversations.

3.4 Episodic Memory Structures

With respect to the theory behind our architectural design, we now analyze the individual components with a high-level discussion of the memory structures. A definition of the requirements, rationale, motivation, and merits of these structures contribute to differentiate our system from previous implementations by other authors. Furthermore, it provides a sound basis on which to discuss memory representation in the episodic memory space.

3.5 Retrieving Episodic Memories

Before episodes become useful to any agent in need of recalling previous experiences, our episodic memory architecture must first be able to select and rank the stored information by its relevance to the particular context. We emphasize the distinction between episode selection and ranking. Selection of episodes calls for browsing stored memories to form a candidate pool of episodes when a request is received by the episodic memory architecture. To support the real-time operation of an ECA, this process must also return results in real time. The selected episodes should form a considerably smaller set than the total episodic memory space. In addition to selecting the episodes, the architecture must assign ranking or confidence levels to each episode as it prepares the set of selected episodes for presentation.

Our implementation utilizes simple mechanisms of episode selection and ranking. The selection of episodes may occur through one of two criteria: (i) temporal vicinity or (ii) contextual relevance. Episodes ranked by temporal vicinity arrange episodes deemed relevant to a context by their relationship in the sequence of events. The reader may visualize, for instance, a request for episodes based on temporal vicinity as a call for episodes that require playback. As a case study, if a user should need information about the presidential elections, then the architecture would first find episodes with relevant information. It then ranks these based on their chronological occurrence in order for the user to receive a temporally ordered view of events. In contrast to the sequential arrangement, our second selection mechanism introduces ranking by contextual relevance. Contextual relevance exploits information gathered as an agent interacts with human(s) to reason about which episodes should be selected for recall. Ranking in searches that perform selection by contextual relevance assign relevance to each episode based on the similarities demonstrated between recalled episodes and the current interaction with the user. The frequency-based algorithm for topic selection that was previously introduced currently serves as the ranking mechanism for contextual relevance.

3.6 Accessing Episodic Memory

In order for an ECA to request information about an episode and trigger the retrieval processes, it must possess the means of communicating with the episodic memory server. Previously, we defined the episodic memory structures and selection methods without considering how the episodes should be accessed. Accessing episodic memory from our architecture occurs through simple public interfaces into the memory.

Our work approximates communication and access models between memory and system components in a similar manner to well-known tools for developing large, complex systems. As a case in point, we allude to Microsoft’s RDS. Microsoft’s RDS permits developers and hobbyists to “create robotic applications across a wide variety of hardware” [34]. It facilitates communication across components by enabling the user to build modules that are loosely coupled [32, 33]. These loosely coupled components can be developed independently and make minimal assumptions about each other or their runtime environment [42].

Per the model of communication that we ascribe to Microsoft RDS, our memory-access interfaces instantiate episodic memory as a service application that uses techniques commonly employed in RDS. However, RDS does not incorporate an episodic memory model on its own. Episodes can be communicated through message-passing mechanisms. More importantly, we attempt to address the requirements in Ref. [33] for Decentralized Software Services (DSS) modules. From the perspective of our memory architecture, DSS modules promote robustness, composability, and observability within complex systems, as follows:

• Robustness – protocols isolate the memory component from the remainder of the system and thus aid in limiting the impact of partial failures arising from episodic memory faults.

• Composability – episodic memory as a service appears to be “created” and “wired up” at runtime based on the system’s needs.

• Observability – it is possible, through our protocols, to determine what the memory is doing, what state it currently holds, and how it arrived at that state.

The episodic memory client is tasked with three responsibilities while in the presence of a dialog: collect user speech and deliver it for storage, recognize user interjections or interruptions, and display episode summaries by retrieving episodic information. Several elements are contained in the structure delivered by the memory interfaces for episode summaries. These elements are as follows:

• episodeGuid – the unique identifier of the conversation being summarized.

• participants – the names of the participants or individuals identified in the conversation.

• timestamp – the time at which the conversation was last updated.

• text – a textual summary of topics relevant to the conversation. At this moment, the summary is provided in a straightforward fashion that identifies the high-level topics first, followed by supporting keywords.

• imagepreviews – a listing of URLs for images that may be representative of themes in the conversation. This aspect is included for demonstrative purposes only and may form part of a future expansion.

Through the data provided by the episode summary interfaces, we populate a chronological view of the dialogs in which the ECA has participated.

Although in the preceding paragraphs we alluded to analysis services, we deferred from exposing the list of services. In the following section, we complete our exposition of the analysis services exploited by the contextualization process.

3.7 Analysis Services

The contextualization process exploits two principal sets of modules: contextualizer methods and analysis services. Contextualizer methods act as interfaces between the database and analysis services. The database itself interacts with the contextualizer by providing notifications of dialog events that, in turn, begin the contextualization process when it receives new user input, or by storing the resulting context segment. Under normal circumstances, the complexity of the operations occurring while the contextualizer awaits analysis services results does not adversely hinder the interaction between the memory interfaces and the database. These interactions occur asynchronously. In other words, the user may continue interacting with the ECA while the backend database services perform contextualization services for a recent bulk of statements.

Analysis services provide the contents of the context attributes tags. In our architecture, we demonstrate the use of up to five services per context segment. Briefly stated, these services stem from several toolkits and APIs:

• SharpNLP – C#-based API used to extract NLP features. It performs tagging, chunking, and parsing of sentences based on a Maximum Entropy algorithm trained on the English Treebank corpus [31].

• AIML – C# implementation of an Artificial Intelligence Markup Language chatterbot based on Program#. It primarily performs template matching on new utterances [50].

• Weka – Java-based API that implements many common classifier algorithms [55].

• OpenCalais – web service that automatically creates semantic annotations for unstructured text based on NLP, machine learning, and other methods [40].

• Yahoo! Search – web service to search web content including news, site, image, and local search content [57].

Only the first of these services above does not provide results to the context attributes tag. Because NLP features are extracted at an earlier stage than that which builds the context attributes, these features are included as part of the resources tag. The remaining four services provide user-model information, domain classification, semantic tagging, and web expansion.

As a variety of user statements can take the form of factoids, we employ an agent based on AIML to identify information in these statements. Program#, our API of choice for this task, can process >30,000 categories in under a second according to Tollervey [50]. Additionally, we can readily expand the AIML domain coverage by augmenting it with template documents. We exploit the template-matching methods to recognize factoids but do not require that subscribing agents act on the suggested response. Instead, the intended benefit from AIML template matching encompasses the ease of updating user models and recognizing factoids.

We implemented a Weka-based service that accepts transcripts as inputs and attempts to classify the type of statement into domain-specific categories. Essentially, it performs analysis for domain-specific applications. In our architecture prototype, we demonstrate the use of domain-specific classifiers by targeting a topic particular to the interests of the US National Science Foundation. The approach of mapping user utterances to likely ECA responses by using domain classification resembles that of Sgt. Blackwell [27].

The remaining two services require an active Internet connection to analyze data. OpenCalais, the first of the external services, performs textual analysis and semantic annotation of the dialogue in progress. OpenCalais [40] identifies a list of 38 entity types and 69 event or fact types that it can recognize. Moreover, it allows us to perform document classification over a range of episode segments in order to classify conversations by their broader topics. Yahoo! Search performs a similar analysis as OpenCalais and identifies certain named entities or key phrases in unstructured text. In addition to this service, it permits us to expand the text and resources available for context analysis by providing an API to perform mining of web documents and multimedia.

4.1 Experiment #1 – Passive Participation

The first set of test measures the relevance to human users of the phrases gisted by our algorithm. We devised a test where the episodic memory system and a human judge act as passive observers of two different prerecorded conversations between an interviewer and an interviewee. Both are human. The episodic memory system “listens” to the conversations and generates key phrases representing the gist of each of the conversations. Textual input and output were used. A GUI later displays the video to nine human judges and presents the topics identified as relevant by the episodic memory system, alongside a Likert scale for each judge to independently assess the relevance of these topics vis-à-vis the video he/she just saw. The human judges thus describe the level of agreement they find with the key phrases generated by the episodic memory model. Zero (0) on the scale denotes a total lack of relationship between the phrases and the video; 10 signifies complete agreement.

To minimize bias, a set of unrelated phrases were included among the topics selected by the episodic memory system. These phrases should be determined by the judges to be of low relevance. At the beginning of each interview, the subject is asked his/her name, followed by a series of background questions. The interviewer is responsible for introducing the topics of conversation. Occasionally, however, the interviewee expands the responses by adding personal experiences to the conversation. Table 2 shows the key phrases detected (gisted) by our episodic memory system.

A total of 202 phrases were presented to the nine test subjects as candidate topics. Of these, 54 phrases corresponded to phrases from the topics gisted by our algorithm (six to each subject), while 148 corresponded to phrases unrelated to the video clips (the noise set).

A statistical summary of the results for the topical phrases is shown in Table 3. As the mean (5.43) suggests, the values selected by the user in response to the gisted phrases indicate that human judges did not generally perceive a strong relationship between the key phrases gisted by the episodic memory system. The range (10) and large standard deviation (3.75) support this assertion.

Table 4 summarizes the same statistics for the noise set. The mean value of the noise set (1.43) is substantially lower than the topical mean (5.43), as was expected. This indicates that the judges correctly saw these as irrelevant.

The mean of the responses from the judges for topical phrases indicates that they are (slightly) biased toward 10, while the responses of the noise set are (strongly) biased toward 0. We use these biases to our advantage by further grouping our data and comparing it to similar works.

Other studies, such as that of Kanayama and Nasukawa [22] and Ku et al. [25], have employed human judges (annotators) as the gold standard for rating opinions or topics extracted from textual corpuses. Ku et al. [25] demonstrate that inter-annotator agreement is about 54% at word level extraction and 52% at sentence level extraction from web blogs.

We compare our results to those of Ku et al. [25] to see whether the agreement demonstrated by the judges and the gisting algorithm resembles the levels obtained from inter-annotator evaluations. To show this, we categorize the user responses into two segments: mostly agree and less likely to agree. The mostly agree category accumulates all the user responses with values >5. Similarly, the less likely to agree category accumulates responses ≤5. Using the Topics set, we are able to obtain a 51.9% agreement between the human judges and the gisting algorithm. Therefore, it follows that our algorithm performs at a level comparable to that of human annotators in the context of the interviews and compatible with the results of Ku et al. [25].

4.2 Experiment #2 – Episodic Memory Integrated into ECA

This experiment sought to evaluate the episodic memory system exactly how we plan for it to be used – integrated into an ECA. Figure 3 displays the graphic representation of the ECA used. We refer to this ECA as the Lifelike Avatar [16]. This experiment compares the relative capabilities of the Lifelike avatar with the episodic memory system incorporated within it to the same avatar without memory. This was done by designing a set of scripts – a sequence of questions that a user might ask. We constructed a total of six scripts that addressed two different knowledge bases. Three of the six scripts addressed a knowledge base about planets – we labeled these scripts P-1, P-2, and P-3. The other three scripts addressed a knowledge base about natural disasters: N-1, N-2, and N-3. Each script contained either six or seven questions. Each of the three scripts for each knowledge base was designed to test a different aspect of the ECA’s performance. Textual input and output were used to avoid the errors introduced by automated speech recognition engines.

Figure 3:

The Lifelike Avatar.

The first scripts (A-1 and N-1) asked straightforward questions that the memory-less ECA should be able to answer correctly. Each question was entirely self-contained, and it was not necessary to infer the meaning of the question from previous contexts in the conversation. The questions in these scripts were control questions used to ensure that our ECA with episodic memory conserved all the functionality of the original (memory-less) Lifelike avatar.

The second scripts (A-2 and N-2) contained questions phrased indirectly that required the use of memory. These scripts focused on a single topic such as asking whether various planets have water. Some of the questions were imprecisely phrased (e.g. “How about Mercury?,” after asking whether there is water on Mars, as opposed to asking “Is there water on Mercury?”). This required the ECA to remember the thread of the conversation.

The last set of scripts (A-3 and N-3) was the most challenging. In these scripts, several questions were also phrased indirectly. However, instead of focusing on a single topic, the topic changes partway through the script. The ECA had to detect the shift in context and respond accordingly. The last question of these scripts was something completely outside of the knowledge base of the ECA (“Do you have any friends that are avatars?”). The ECA should recognize that it does not know the answer, and respond accordingly. The scripts also contained questions that were repeated multiple times at different points in the conversation. The ECA should indicate in some way that it was aware of the repetition.

We analyze the results by placing each question asked into a category describing the skill that it was designed to test. The categories are as follows:

• Direct: the question is entirely self-contained.

• Reference: interpretation requires using memory about previous events in the conversation.

• Repeat: the question was asked previously in the script. These questions were also counted under another category that measures whether the content of the ECA’s answer is correct. The “repeat” category was used to account for awareness of repetition.

• Unknown: the question is outside the scope of the knowledge base. The ECAs answer should indicate that it does not know the answer.

The answers had to be correct. Given that the questions were relatively simple and unambiguous (“Is there water on Mars?”), the answers were judged as correct/incorrect by the testers themselves, based on what is contained in the knowledge base. Furthermore, if the question was repetitive, the ECA had to respond in a way that recognized that this question had already been answered (“I already told you that Mars has no water”). We considered the performance of each ECA (with and without episodic memory) to be the total number of questions answered correctly, summed across all categories, divided by the total number of questions. The performance breakdown by categories and the total performance is shown in Table 5.

The overall performance of the ECA equipped with our episodic memory was substantially superior to the ECA without memory. The results show that the addition of our memory architecture conserves the strength of the original Lifelike avatar – its ability to correctly parse direct, self-contained queries. In this category, both ECAs achieved an equal perfect score. The advantage of the ECA with our memory architecture comes in the categories that require use of context from elsewhere in the conversation. Here, we see that the ECA with memory was able to correctly answer every question, while the one without memory was not able to correctly answer any at all.

Interestingly enough, we did observe a performance decrease for the ECA with memory when asked unknown questions. All three failures occurred with the natural disasters knowledge base. Essentially, the additional information to which the ECA-cum-memory had access led it to find false positives, where it thought that the query actually matched some answer in its knowledge base. This issue is particular to that knowledge base and phrasing used: it never occurred in the planets knowledge base and did not occur when a different (but equally unknown) question was asked. Eliminating false positives generated by the spurious use of context represents an area for future work.