Non-monotonic Generation of Knowledge Paths for Context Understanding

A knowledge graph represents human knowledge in the form of semantic entities and relations among these entities. In recent years, knowledge graphs have been applied to many information system applications, including decision support systems (DSS) [24, 36, 41, 67], information retrieval (IR) [4, 17, 34, 62], question answering (QA) [49, 61], and recommendation [8, 12, 57, 71] to help with deriving or explaining the results of these applications. Meanwhile, industries and user communities are also developing large knowledge graphs for commercial and public use. For example, Google’s knowledge graph, which contains hundreds of millions of entities, is being used for improving the search of people, locations, and other interesting objects.¹ DBpedia is another knowledge graph constructed by the Wikipedia user community and it contains at least 6 million entities [50].

To illustrate how one leverages a knowledge graph for information retrieval, we consider the task of identifying fake news among a collection of news articles [19, 22]. In a recent MIS work on fake news identification [19], the trustworthiness of each third party digital supply chain used by online news websites has been determined to be a useful feature to predict if a given online news article is fake. Another feature that aids fake news prediction is the checking of facts mentioned in the news article. As a knowledge graph provides entities and relations possibly mentioned by a news article, verifying the facts in a news article against knowledge graph entities and relations is thus a good strategy to improve the fake news prediction accuracy. By performing the same verification for a sampled collection of news articles from the same digital supply chain against the knowledge graph, one may also calibrate the supply chain’s trustworthiness more accurately. Once a fake news article is detected, one can provide warnings or interventions to help users make appropriate judgments about the article content [7].

Despite the several useful knowledge graph applications, many existing knowledge graphs are incomplete as they may not be up-to-date with the latest entities or relations [10, 65]. Hence, many research projects on knowledge graph completion, i.e., identifying missing entities, entity attributes, and relations, have been carried out lately [45, 52]. The earlier works in this domain focus on predicting missing relations from observed entities and relations [5, 33]. However, such methods fail to handle unseen entities, or paths of relations that semantically connect two entities of a knowledge graph. As a result, some research works turn to knowledge graph completion with input unstructured data such as text [44, 60]. As part of information systems research, there are several research efforts focusing on knowledge extraction and construction [11, 51, 63]. Specifically, Gil et al. designed software systems to extract knowledge from different software artifacts (e.g., purpose, features, information, deployment strategy) [20]. These methods learn to infer new relations from input textual information, and use them to augment the knowledge graph. Still, most of these works focus on extracting one-hop relations from text instead of paths made up of entities and relations.

For the above example, the existing methods that infer one-hop relations would fail to recover the two relations \(e_{\text{Daniel Craig}}\xrightarrow {{\bf starring}}e_{\text{Casino Royale}}\) and \(e_{\text{Casino Royale}} \xrightarrow {{\bf director}}e_{\text{Martin Campbell}}\) , as Casino Royale occurs very far from both Daniel Craig and Martin Campbell in the news text.

In this article, we therefore aim to infer a path of relations from a piece of input text capturing the semantic connection between two given entities. This inference is performed based on an incomplete knowledge graph, and such an inferred path is called contextual path. For the example shown in Figure 1(a), the two given entities are Daniel Craig and Martin Campbell and the input text is the news article in Figure 1(a). The input knowledge graph is shown in Figure 1(a) and (b). The contextual path inferred is \(e_{\text{Daniel Craig}}\xrightarrow {{\bf starring}}e_{\text{Casino Royale}} \xrightarrow {{\bf director}}e_{\text{Martin Campbell}}\) . In this example, the two relations in the contextual path happen to exist in the knowledge graph. In general, we would also like to infer new relations for the knowledge graph when they are required to construct contextual paths.

Suppose the contextual paths connecting between many different pairs of entities mentioned in a collection of news articles are inferred or generated; the paths and their relations can support many new interesting ways to search and browse the articles. For instance, one can easily search news articles mentioning actors or actresses who star in Martin Campbell’s movies.

In the above example, we assume that text data such as news articles have their entity mentions extracted and linked to the corresponding entities of the given knowledge graph. Among the methods that recognize such entity mentions, named entity recognition methods help to detect person names, organization names, place names, and other names in textual content followed by linking these entity mentions with entities of the knowledge graph (also known as entity linking) [42, 43]. Entity linking methods perform the matching of mentions in textual content with knowledge graph entities.

In this article, we therefore focus on addressing the following question: “how exactly do two entities connect to each other given a textual content as the context?” Context is crucial in this question as the same entities can be semantically connected differently in different contexts. To answer the above question, we formulate the contextual path generation problem.

Finding the contextual path is challenging for a number of reasons. First, instead of returning all paths that connect \(e_h\) and \(e_t\) , CPG has to determine the path that is most relevant to the context d, including the involvement of entities not mentioned in the context. We call this the “path relevance” challenge. This challenge is largely caused by noisy and sparse data in the context, as well as the implicit patterns of constructing relevant contextual paths. The former is caused by other entities mentioned in the context but may not be relevant, and absence of entities not mentioned in the context but may be relevant, which could be due to very little content in the context. The latter refers to the expected pattern of contextual path for \(e_h\) and \(e_t\) of specific types. For example, a contextual path linking an actor entity with a director entity via a movie entity is certainly more plausible than via a concert entity. Second, the given knowledge graph may not cover all the required relation edges to form the contextual path. We call this the “incomplete knowledge graph” challenge. Given an incomplete knowledge graph, we have to generate contextual paths in CPG instead of constructing paths using observed relation edges. In the generation process, we face the third challenge of ensuring the resultant paths to be of good quality. This is also known as the “path well-formedness” challenge. Finally, as there are no prior works on CPG, we also need to address the “data challenge” of constructing datasets for training and evaluating any proposed CPG solution models.

This article therefore aims to solve the novel CPG problem while addressing all the above challenges. We propose a general two-stage framework for solving CPG. We then use that to develop solution models. In the following, we outline our research contributions:

We organize the rest of this article as follows. We present our literature review in Section 2. Section 3 covers our proposed Two-Stage framework and our proposed Non-Monotonic CPG model. We construct the two real-world datasets and introduce the evaluation metrics specially designed for CPG in Section 4. In Section 5, we cover the experiment setup and results/findings. Finally, Section 7 concludes the article and outlines the future research directions.

Using knowledge graphs for enhancing information systems has been an active research area in information systems [9, 16, 69]. The common goal of these works is to capture and employ existing human knowledge to improve the functional features of information systems. For instance, Liu et al. derived inter-company connections from relations of an enterprise knowledge graph and utilized these connections to derive for a target company news sentiment features of connected companies together with news sentiment features of the target company in order to predict the target company’s stock movement [37]. Lim et al. proposed to improve the accuracy of adverse drug reaction prediction by learning better vector representation of clinical concepts and relations in a knowledge graph using the confidence weights assigned to them based on co-occurrence and NLP patterns found in medical literature [30]. Xu et al. proposed to measure social proximity between two company entities using the similarity between the embeddings of the two entities in the knowledge graph [64]. CPG can be applied to the above company and medical domains to return contextual paths between entities (e.g., companies or medical concepts) mentioned in text collection (e.g., news articles and medical literature). These contextual paths can be used to establish new inter-entity connections or to derive confidence weights between entities so as to improve prediction accuracy or social proximity measurement.

Finding a contextual path is somewhat similar to reasoning with knowledge graphs. The previous knowledge reasoning works adopt a wide range of reasoning techniques, including logical rules [2, 26], Bayesian models [59], distributed representations [47, 55], neural networks [14, 25, 29, 32], and reinforcement learning [13, 21, 28, 35].

Knowledge reasoning has been widely applied to knowledge graph–based question answering (QA). In QA, Asai et. al proposed a retriever-reader framework on the HotpotQA dataset that reasons over Wikipedia using its graph structure [1]. While the HotpotQA dataset itself, however, does not include a knowledge graph, it can be linked to the Wikipedia knowledge graph for explaining the questions’ answers. MetaQA traverses through a knowledge graph in order to retrieve the answer to a given question [68]. Although the traversed path could explain the answer, the QA task itself is not designed to evaluate the traversed path against a human judged explanation path [31, 61]. By definition, traversed path can only be constructed with observed relations of the knowledge graph. MHQA-GRN addresses reading comprehension QA by constructing a directed acyclic graph from a knowledge graph for each passage of the query article and using its representation as features to predict the answer [54]. Nevertheless, MHQA-GRN does not return a path similar to contextual path. LEGO is a framework designed to find the answer entities for a given question by alternating between generation of query tree and reasoning with latent space [49]. Starting from the question entities as the root, LEGO expands the query tree with extracted relations from the question. In the representation space, the projection of the relations form several candidate answer spaces near the question entities. LEGO then finds the answer entities lying within the intersection of the candidate answer spaces. In [73], researchers combined a convolutional neural network for feature extraction with a recurrent neural network to predict the answer entity given a query entity and a knowledge graph that covers different paths from the query entity to candidate answer entities. CogKR conducts multi-hop reasoning over a knowledge graph by iterating between an expansion module and a reasoning module to predict the answer entity for a given query entity using a neighborhood knowledge subgraph [18].

In summary, we found that none of the above models considers context documents when selecting the knowledge paths leading to the answer entities. They also do not perform any evaluation on the knowledge paths derived as a side result. Hence, our CPG work can contribute to existing knowledge reasoning works by bridging these two research gaps.

Both the path relevance and the well-formedness challenges require us to generate well-formed contextual paths relevant to the input context document and query entity pair. We can thus treat CPG as a conditional sequence generation task with the context document and query entity pair as the input condition.

Our survey found that most conditional sequential generation works focus on generating text conditioned on a simple label such as tense, sentiment polarity, or formality of the text to be generated [15, 23, 53, 66]. They could not cope with the complex input condition of CPG that involves entity and text data. Among the few recent works that consider text as input condition, BERT-fused translation uses a pretrained encoder to first obtain the representation of text condition before performing text generation [72]. KG-BART takes a set of concepts from a commonsense knowledge graph as input and generates a piece of text covering these input concepts [39]. Other related works include generating sentences conditioned on a text document following a given syntax of an exemplar [27, 46]. So far, all these works focus on generating natural language sentences instead of knowledge paths. To our knowledge, there is so far no research focusing on generating knowledge paths with complex input conditions.

Unlike the above works, we propose to learn a pretrained model to generate knowledge paths before fine-tuning it to generate conditional knowledge paths. Our research also focuses on making the paths well-formed as well as semantically relevant to the query and context document.

Before we present the framework, we give the definition of knowledge graph. We define the knowledge graph G to be a tuple \((E,L,R)\) where E denotes a set of entities, L denotes a set of relation edges, and R denotes a set of relation labels or simply relations. Each relation edge \((e_i,r_k,e_j)\) of L directly links entity \(e_i\) to entity \(e_j\) via a relation edge with label \(r_k\) that can be found in R. We can also denote the same edge by \(e_i \xrightarrow {r_k} e_j\) . Notation wise, we use italic for entities (e.g., Biden) and boldface (e.g., presidentof) for relations. For the purpose of establishing connections between entities, we assume that each relation and its reverse can always be found in R. For example, for the knowledge graph to capture Biden is the president of USA, R should contain both \(e_\text{Biden} \xrightarrow {{\bf presidentof}} e_\text{USA}\) and \(e_\text{USA} \xrightarrow {{\bf presidentof}^{-1}} e_\text{Biden}\) .

The input of CPG can be captured as a query q represented by \((e_h, e_t, d)\) where \(e_h\) and \(e_t\) are the input entities and d is the context document. The output of CPG is a path p represented as a sequence of path elements denoted by \(\langle e_{q,1}, r_{q,1}, e_{q,2},r_{q,2},\ldots ,\) \(r_{q,(n_{q}-1)},\) \(e_{q,n_{q}} \rangle\) where \(e_{q,1}=e_h\) and \(e_{q,n_q} = e_t\) . The path elements include the entities and relation labels involved in the path. We assume that all entities and relation labels that are used in any path should be found in E and R, respectively, i.e., \(\forall i, e_{q,i} \in E\) and \(\forall k, r_{q,k} \in L\) . Nevertheless, it is possible that some relation edges of a path may not exist in the knowledge graph G as the latter is incomplete.

As shown in Figure 2, our proposed framework consists of a context extractor and a contextual path generator. The former derives context entities \(E(q)=\lbrace e_1, e_2,\ldots ,e_{ \vert E(q) \vert }\rbrace\) from the query q and input knowledge graph G. Context entities are entities relevant to the query and they may be mentioned in d. To enrich the sparse content of the context document, one should also select relevant entities from the knowledge graph to be context entities thereby addressing part of the path relevance challenge (as the remaining part is to be addressed by the contextual path generator).

The context entities together with the query are then provided to the contextual path generator to construct the resultant contextual path(s) leveraging the interaction between context entities to finally determine the relevant entities and relation edges and sequence them to form the resultant path. In this process, new relation edges may be inferred to address the incomplete knowledge graph challenge. We further propose a non-monotonic path generation process to ensure that the resultant path is both semantically relevant and well-formed addressing the path relevance and path well-formedness challenges, respectively.

The above non-monotonic path generation approach is novel and is different from the existing single object-type monotonic sequence generation approach, considering that a knowledge path is a type of sequence. Single object-type sequence generation is one that generates elements of the same type, instead of entities and relation edges as alternating path elements. The monotonic sequence generation is one that always generates elements from left to right only. Such generation may fail to reach the tail entity \(e_t\) , and repeats the same path element(s). In this case, the generated path is not well-formed. As non-well-formed paths are considered incorrect, our goal is thus to design a generation model that always generates well-formed contextual paths.

The context extractor takes a query \(q=(e_h, e_t, d)\) and a knowledge graph G as input, and decides the set of context entities \(E(q)\) . Ideally, \(E(q)\) contains the set of entities for constructing the correct contextual path. In this section, we propose two context extraction methods: Knowledge-enabled Embedding Matching Model (KEMatch) and Learning-To-Rank with Multi-Head Self-Attention Model (LTRMHSA). The former is based on binary classification, and the latter is based on learning-to-rank on multi-head self-attention.

The two methods share a common Query-induced Entity Selector that derives a subset of knowledge graph’s entities \(E^r(q)=\lbrace e_1, e_2, \ldots , e_{m_q}\rbrace\) as candidate entities using the query entities \(e_h\) and \(e_t\) where \(m_q = |E^r(q)|\) .

Let \(E^r(h)\) and \(E^r(t)\) be the sets of entities in the knowledge graph G that are reachable from \(e_h\) and \(e_t\) , respectively, in k-hops. The set of candidate entities is then obtained by \(E^r(q)=E^r(h)\cup E^r(t)\) . The remaining modules of KEMatch and LTRMHSA then select a subset of entities from \(E^r(q)\) to form the context entity set \(E(q)\) .

CPG is responsible for generating a path \(p_q\) given \(E(q)\) and for inferring missing relations when needed. We propose a Non-Monotonic Contextual Path Generation with Pretrained Transformer (NMCPGT) method as shown in Figure 4. NMCPGT addresses the two limitations of traditional monotonic generation models such as n-gram [3] or neural language models [48] where the order of element generation is always from left to right. The first limitation is the generation of unfinished paths when the model generates a path that starts with \(e_h\) but could not end with \(e_t\) . The second limitation is the difficulty to leverage on both \(e_h\) and \(e_t\) to determine the next element to generate in the monotonic manner. Thus, in this article, we propose the non-monotonic path generation model that substantially increases the odds of finished paths and is trained to generate the more likely path elements using \(e_h\) , \(e_t\) , and the other already generated path elements in each generation step.

NMCPGT is obtained in two steps, namely, the pretraining and fine-tuning steps. In pretraining, a pretrained path generation model is trained to be familiar with the knowledge graph’s entities and relations and the ways entities of different types are connected with one another via relation edges, and it is able to generate knowledge paths given an \((e_h,e_t)\) pair. In fine-tuning, we further train the model to include candidate entities \(E(q)\) as additional input and to return the contextual path.

During both the pretraining and fine-tuning phases, we need to feed the initial generation state represented as a binary tree to the transformer for training. Since the transformer can only generate a sequence of objects, it is trained to decode or generate the sequence of actions that represents the serialized binary tree of the training path. We serialize a path by traversing it in a top-down, left-to-right manner. The serialization process is shown in Figure 6(a). For each timestep t, the transformer decoder generates (or predicts) the next action of binary tree, which is in turn used as input to the transformer decoder for generating the next action. This repetitive process ends when the binary tree is terminal.

In the absence of suitable datasets, we constructed two datasets, Wiki-film and Wiki-music, each consisting of 40 Wikinews articles as context documents. We then identify 563 and 471 entities mentioned in these two sets of context documents, respectively, which can be found in a knowledge graph extracted from DBpedia.² In the DBpedia knowledge graph, each entity corresponds to an article in Wikipedia. Every attribute within the infobox of the article that refers to another article is extracted as a relation edge linking the entity of the first article to the entity corresponding to the second article. The attribute label in the infobox is used as the relation label. To derive the set of \((e_h,e_t)\) entity pairs, we identify 1,396 and 1,237 entity pairs from the context documents of the Wiki-film and Wiki-music datasets such that both entities of each pair can be found in a context document with paths connecting them in the knowledge graph. The detailed statistics of the datasets are shown in the second and third columns of Table 1.

We crowdsourced the ground truth contextual path for each \((e_h, e_t)\) entity pair using AMT workers annotating the path using a custom-developed web-based annotation interface. To make the annotation reasonable to most annotators, we limit the contextual paths to have a maximum length of six. At the end of crowdsourcing, we derive one ground truth contextual path for each \((e_h,e_t)\) pair with majority agreement.³

To measure the CPG performance, we divide the \((e_h,e_t)\) entity pairs of each dataset into five folds. Each fold takes a turn to be used for testing while the remaining four folds are used for model training. We repeat this process five times and report the averaged performance across five folds in the experiment. We measure the model performance using three types of metrics, namely, (a) percentage of recovered ground truth paths; (b) averaged pairwise similarity; and (c) normalized graph edit distance.

Percentage of recovered ground truth paths (%Path Recovered). This metric measures the proportion of generated paths that are identical to the ground truth path. In this metric, the similarity between the generated and ground truth paths is not considered.

Average pairwise similarity (AVG PW Sim). This metric shows how much a generated path overlaps with the ground truth contextual path. Given a path p generated by the model, we compute its pairwise similarity with the ground truth path \(p^*\) bywhere \(E(p)\) and \(L(p)\) represent the set of entities and relation labels existing in p, respectively. We then compute the average pairwise similarity for all the generated paths. While this metric can effectively determine how close the generated path is to ground truth, it does not consider the ordering of the entities and relations. Moreover, it does not take into consideration the semantic relatedness between the two paths. The next normalized graph edit distance metric thus addresses these limitations.

Normalized Graph Edit Distance (NGEO). The Graph Edit Distance (GEO) is a metric originally designed to measure the similarity of two graphs by counting the number of operations needed to transform one graph to another [70]. We adapt it to measure the similarity between a generated element sequence q and the ground truth element sequence \(q^*\) . In this work, we report the normalized GEO:where \(\vert q^* \vert\) is the length of the ground truth sequence. From the full path ( \(e_1,r_1, \ldots ,r_{L-1},e_L\) ), we derive an entity sequence where only entities are included in a path ( \(e_1,..,e_L\) ), and a relation sequence ( \(r_1, \ldots ,r_{L-1}\) ) defined in a similar manner. We denote the NGEO for relation and entity sequences as NGEO(E) and NGEO(R), respectively.

Let \(OP(q, q^*)\) be the sequence of edit operations for converting the former to the latter. The GEO metric is defined by

There are six operations defined: entity insertion, entity deletion, entity substitution, relation insertion, relation deletion, and relation substitution. The semantic cost of an operation is defined by entity type or relation label distance determined by the ontology structure underlying the entities and relation labels. GEO makes use of the ontology structure of the knowledge graph to determine the semantic similarity between entities and relations. Let t (or r) and \(t^{\prime }\) (or \(r^{\prime }\) ) be the inserted entity’s type (or inserted relation label) and the deleted entity’s type (or deleted relation label), respectively. \(t_0\) and \(r_0\) are the root entity and relation, respectively, in the knowledge ontology. For each operation op performed on a path, the semantic cost \(c_{op}(\cdot)\) incurred is defined below:

When a semantic operation is performed, the semantic distance is defined as the co-topic distance \(dist(t_1.t_2)\) (or \(dist(r_1,r_2)\) ) of the two ontological types \(t_1\) and \(t_2\) (or relation labels \(r_1\) and \(r_2\) ) in a knowledge graph ontology:where \(S(t_i)\) is the set of supertypes of \(t_i\) in the ontology. For example, given an ontology consisting of supertype-subtype relations, \(\lbrace\) Agent \(\rightarrow\) Person, Person \(\rightarrow\) Artist, Artist \(\rightarrow\) Actor, and Person \(\rightarrow\) MovieDirector \(\rbrace\) , the distance between Actor and MovieDirector is \(1-\frac{2}{5} = \frac{3}{5}\) .

Our first set of experiments evaluates the effectiveness of context extraction models returning the ground truth contextual path entities as context entities for a set of queries. Other than our proposed context extractor methods KEMatch and LTRMHSA, we also include a context window baseline method described below.

We use a pretrained DBpedia k-bert model as the context encoder [38]. The model uses the following configuration: \(L=12\) , \(A=12\) , \(H=768\) .

We measure the performance of context entity extraction by precision and recall defined by Precision \(=\frac{|E(q)\cap E^*(q)|}{|E(q)|}\) and Recall \(=\frac{|E(q)\cap E^*(q)|}{|E^*(q)|}\) , respectively. \(E(q)\) denotes the set of context entities extracted by the extractor and \(E^*(q)\) is the set of ground truth context entities. We utilize the negative sampling process described under Section 3 and conduct 10-fold cross validation. As we have consistent observations on Wiki-film and Wiki-music and due to limitations in space, we only show the experiment result of the Wiki-film dataset in Table 2. In this comparison, we additionally include the context window baseline.

LTRMHSA, which considers inter-entity interaction, yields the best precision (85.9%) and recall (83.1%). KEMatch using contextual query entity representation with the Hadamard product feature combination option, returns the next best precision (81.3%) and recall (78.9%). This is followed by KEMatch using mention paragraph representation also with the Hadamard product feature combination option. These results suggest that the contextualized query entity representation is the best option for KEMatch. Finally, the context window baseline is the worst-performing method as it does not involve the knowledge graph entities.

Among the other KEMatch variants, average entity representation performs better than title and mention paragraph representation, possibly due to the latter’s high feature dimensionality. We observe features with a high coefficient have been assigned to both \(\mathsf {p}_1\) and \(\mathsf {p}_q\) . This observation supports our hypothesis that the title paragraph contains useful information about the contextual relationship between the query entities. Finally, among the different feature representations, the Hadamard product achieves the best performance, followed by concatenation and subtraction. Although all three of these feature representations show decent predictive results, combining them together does not result in better performance due to high feature dimension.

While Table 2 shows that LTRMHSA enjoys higher accuracy than all of the KEMatch variants, the former involves a computational complexity of \(O(m_q^2)\) compared with KEMatch’s \(O(m_q)\) . When \(m_q\) is large, the computational overhead of LTRMHSA could be significantly higher, which adversely affects the time needed for generating a contextual path. As a result, one should take into consideration the tradeoff between performance and execution time when deciding which context extractor to use together with NMCPGT in practice.

Next, we conduct experiments to evaluate the models’ performance in generating the contextual paths. We compare our proposed models combining NMCPGT with different context extractor options, namely, KEMatch and its variants, LTRMHSA, context window entities, and random context entities. The random context entities method randomly selects \(n^{CXT}\) entities from \(G(q)\) as the context entities. We set \(n^{CXT}=10\) , which is consistent with the \(n^{CXT}\) for KEMatch and LTRMHSA. For simplicity, we name our models by “NMCPGT \(+\langle\) context extractor method \(\rangle\) ” where the \(\langle\) context extractor method \(\rangle\) options are Random Context, Context Window, KEMatch, and LTRMHSA.

To evaluate the importance of the context document, we additionally include a non-contextual non-monotonic path generation (NCNMPG) baseline that generates the contextual path from path generator that only takes \(e_h\) and \(e_t\) as input, without any knowledge of the context document. In other words, this model does not use any context entities as condition, and only relies on the pretrained transformer when generating paths.

For the path generation, we only include the annealed coaching oracle \(\pi ^*_{annealed}\) as it has been shown to outperform the uniform and coaching oracles in [6]. The model is optimized with cross entropy loss. The non-monotonic path generator uses a four-layer transformer structure with four attention heads, hidden dimension of 256, and feed-forward dimension of 1,024. Adam optimizer is used with the initial learning rate of \(10^{-{5}}\) . The model is trained with 100 epochs. After the 50 burn-in epochs, \(\beta\) is linearly annealed from 1.0 by 0.01 in each epoch. The implementation of all neural structures is based on Pytorch.

We report the % ground truth recovered (%Recov), averaged pairwise similarity (AVG PW Sim), NGEO(E), and NGEO(R) of the generated contextual paths by different models in Table 3. We have similar observations based on the experiments on the two real-world datasets. The CPG performance is also consistent with the results in Table 2. The model NMCPGT+LTRMHSA outperforms other models for all metrics, followed by NMCPGT+KEMatch with context query entity representation and mention paragraph representation. Among the NMCPGT+KEMatch model variants, the Hadamard product is the best feature combination method while subtraction is the worst. The simple baselines that do not extract context entities, including Non-Contextualized Generation and Random Context baseline, perform the worst. This indicates the importance of a good context extractor—the better we extract context entities, the more accurate the generated contextual paths will be. In the following, we use two case examples to illustrate the difference between the different models.

Case example 1. Consider the query entities Zacharias Kunuk and Inuit in the following context from Wiki-film:

The ground truth contextual path is \(e_\text{Kunuk}\xrightarrow {{\bf director}}e_\text{TheJournalOfKnudRasmussen}\xrightarrow {{\bf pageLink}}e_\text{Inuit}\) , which is successfully generated by both NMCPGT+LTRMHSA and NMCPGT+KEMatch with context query entity representation. NMCPGT+KEMatch with mention paragraph representation, on the other hand, generates \(e_\text{Kunuk}\xrightarrow {{\bf producer}}e_\text{TheJournalOfKnudRasmussen}\xrightarrow {{\bf pageLink}}e_\text{Inuit}\) . Although this is not a ground truth path, it is semantically correct as Kunuk is the co-founder of the production company Igloolik, Nunavut company. Both NMCPGT+context window and NMCPGT+KEMatch with AVG entity representation extractors generate \(e_\text{Kunuk}\xrightarrow {{\bf director}}e_\text{Atanarjuat}\xrightarrow {{\bf pageLink}}e_\text{Inuit}\) , which suggests the generation of path is affected by entities within the context document. Finally, the non-contextualized generation and NMCPGT+random context baseline generates the shortest path between the query entities in knowledge graph: \(e_\text{Kunuk}\xrightarrow {{\bf pageLink}}e_\text{Inuit}\) .

Case example 2. Consider query entities James Bond and Pinewood Studio in the following context from Wiki-film:

Both NMCPGT+LTRMHSA and NMCPGT+KEMatch with context query entity representation generate \(e_\text{JamesBond}\xrightarrow {{\bf isSeriesOf}}e_\text{Dr.No}\xrightarrow {{\bf pageLink}}e_\text{PinewoodStudio}\) , which is different from the ground truth path \(e_\text{JamesBond}\xrightarrow {{\bf isSeriesOf}}e_\text{CasinoRoyale}\xrightarrow {{\bf pageLink}}e_\text{PinewoodStudio}\) . Nevertheless, the path is considered semantically correct as the event carried by both paths exists in the context document. NMCPGT+KEMatch with mention paragraph representation successfully generates the ground truth path as it only focuses on the paragraph in which Casino Royale is mentioned. It is not affected by the mention of Dr. No.

In this section, we compare our non-monotonic path generator NMCPGT+LTRMHSA with a left-to-right counterpart by evaluating them on the Wiki-film dataset. As discussed in Section 3.3, we choose to use a non-monotonic generation model because left-to-right generation models are likely to generate unfinished paths.

To conduct this experiment, we replace \(\pi ^*_{annealed}\) of NMCPGT by a Left-to-Right Oracle \(\pi ^*_{L\text{-}R}\) , which always assigns probability of 1 to the leftmost to-be-generated path element of the sequence. As suggested in [6], \(\pi ^*_{L\text{-}R}\) results in maximum likelihood learning of an autoregressive sequence model, which makes the generation process identical to neural sequence models such as GPT-2 [48]. We use the same input sequence, “[soc] \(e_1\) \(e_2\) ... \(e_{\vert E(q)\vert }\) [sep] \(e_h\) [e] \(e_t\) [e] [eoc]”, for fine-tuning the non-monotonic and monotonic models. In this experiment, we include a new metric representing the percentage of %unfinished path in the result (%Unf) to evaluate the model’s ability to complete the path. We define an unfinished path as one that starts with \(e_h\) but could not end with \(e_t\) within \(L_{MAX}=6\) relation edges. Note that the generated paths, finished or not, may involve inferred relation edges. We report the experiment result on the Wiki-film dataset for NMCPGT+LTRMHSA in Table 4. Since NMCPGT+LTRMHSA is required to generate path elements between \(e_h\) and \(e_t\) , it has %Unf = 0. The monotonic model, however, sees 29.73% of the generated paths unfinished. With fewer finished paths, the %path recovered of the monotonic model is also substantially less than NMCPGT+LTRMHSA.

While the monotonic model performs badly in well-formed path generation, it achieves decent average pairwise similarity and NGEO results. This suggests that the monotonic model generates entities and relation labels that are still relevant to the context with the help of LTRMHSA context extractor. For instance, for the query with entities ( \(e_{\text{Brokeback Mountain}}\) , \(e_{\text{Mel Gibson}}\) ) and the following context document:

The ground truth contextual path of this query should be \(e_\text{BrokebackMountain}\xrightarrow {{\bf starring}}e_\text{Ledger}\xrightarrow {{\bf starring}}e_\text{ThePatriot}\xrightarrow {{\bf starring}}e_\text{Gibson}\) . The non-monotonic model using \(\pi ^*_{annealed}\) generates the ground truth path perfectly. On the other hand, the monotonic model using \(\pi ^*_{L\text{-}R}\) generates a path that is unfinished: \(e_\text{BrokebackMountain}\xrightarrow {{\bf subject}}e_\text{AcademyAward}\xrightarrow {{\bf subject}}e_\text{AcademyAwardForBestActor}\xrightarrow {{\bf subject}}e_\text{Ledger}\xrightarrow {{\bf starring}}e_\text{ThePatriot}\xrightarrow {{\bf pageLink}}e_\text{Ledger}\) . Nevertheless, the generated path is still very relevant to the context. In fact, if we do not force the model to stop at the length of 6, it will eventually reach \(e_{\text{Mel Gibson}}\) at the 8th hop. This example explains why we obtain relatively high pairwise similarity and NGEO for the monotonic model.

We conduct the evaluation of selected models and baselines with the synthetic dataset, as shown in Table 6. The results are generally consistent with those in the Wiki-film and Wiki-music datasets. However, we observe that the performance results of all models are poorer in the synthetic dataset. For example, the % Path Recovered of NMCPGT+LTRMHSA is about 10% less than that of the same model on Wiki-music. This may be due to the additional noises introduced to the contextual documents and paths when constructing the synthetic dataset.

As the synthetic dataset is significantly larger, we also examine the efficiency of our proposed model. We conduct the experiments on a Tesla V100 GPU server with 32 GB memory. With 4,000 training instances, the training process costs 1 h 20 min 34 sec. The generation of contextual path for a test query requires around 0.86 second. This result shows the ability of our two-stage model in handling large datasets.

To evaluate the models’ abilities to infer edges that are not observed in the current knowledge graph, we randomly drop k% of the edges from the knowledge graph of the synthetic dataset. We report the performance of MNCPGT+LTRMHSA only, our best performing model, in Table 7. In addition to % path recovered (%Recov), AVG pairwise similarity (AVG PW Sim), and normalized graph edit distance (NGEO), we also report % \(E^-_{\text{CP}}\) Recov, which represents the percentage of ground truth path edges that are recovered.

The results show that as k increases, our model performs poorer by all metrics. When k is small (e.g., \(k=10\) ), the model successfully recovers 65.92% of the ground truth contextual path edges that are removed from the knowledge graph. When \(k=50\) , the %Recov performance drops significantly to 47.37% but the model still recovers 40.85% of removed ground truth edges. Empirically, we also observe that NMCPGT+LTRMHSA is more likely to recover edges between entities that are logically or semantically related. For instance, we find that a movie m is inferred to be made in country c (i.e., \(e_m\xrightarrow {{\bf country}}e_{\text{France}}\) ) if it mostly stars actors and actresses from c. As a result, the model successfully recovers \(e_{\text{Top Gun}}\xrightarrow {{\bf country}}e_{\text{United States}}\) , and \(e_{\text{Amour(film)}}\xrightarrow {{\bf country}}e_{\text{France}}\) . On the other hand, the model fails to recover edges like \(e_\text{Ridley Scott}\xrightarrow {{\bf producer}}e_{\text{American Gangster(film)}}\) , \(e_{\text{Crimson Tide}}\xrightarrow {{\bf starring}}e_{\text{Viggo Mortensen}}\) , and \(e_{\text{Clint Eastwood}}\xrightarrow {{\bf artist}}e_{\text{Bar Room Buddies}}\) . These are edges with specific semantics and thus more difficult to infer.