Contextual Path Retrieval: A Contextual Entity Relation Embedding-based Approach

With knowledge graphs covering many different topical domains, mentions of entities in documents can be linked to knowledge graphs. Knowledge graph-assisted question-answering and exploratory search are becoming very important. These applications typically involve different types of retrieval tasks, querying, or extracting pieces of knowledge from the knowledge graphs. Knowledge graph completion task, for example, is a kind of knowledge retrieval task that predicts entities connecting to a given query-entity via a specific relation. Knowledge-based question-answering is another task that aims to answer factual questions by searching relations in a knowledge graph connecting some question entity to the answer entities [27, 49, 52, 57]. Both tasks address the “what” questions (i.e., what are the possible items that could be recommended to the user?) but not the “how” and “why” questions (i.e., how does this item appeal to the use? Why should we recommend this item to the user?). To cope with the latter, we need a different type of knowledge retrieval task that returns relevant parts of the knowledge graph as answers or part of answers.

In this article, we thus propose a novel knowledge retrieval task called contextual path retrieval. Unlike knowledge-based completion and question-answering, contextual path retrieval requires a good understanding of the query context before the correct answers can be returned. The query in this case is a pair of input entities mentioned in some document, and the answer is a path connecting the entities extracted from the knowledge graph. To the best of our knowledge, this task has not yet been studied, and it represents an early attempt to mimic the human wisdom in establishing contextual connections between two entities. We shall elaborate on the task definition below.

Contextual path retrieval (CPR) is defined as the task of finding path(s) between a pair of query entities in a knowledge graph to explain the connection between them when they appear together in a given context. In this definition, the two query entities form the input query and the result path is invariant to the query entity order. We define the context to be a document covering some common topic or event. The knowledge graph contains the entities and relations from which every result path is to be retrieved.

Example: Consider this query: “Why is Roger Moore related to Daniel Craig in the context of a movie entertainment news article?” Roger Moore and Daniel Craig are the two input query entities, and the movie entertainment news is the context. Here, the background knowledge graph is assumed to be movie related entities and relations extracted from DBpedia. For this pair of query entities, the correct result is a path with two relations, \({\it Roger Moore}\ \xleftarrow {{\bf Portrayer}}\ {\it James Bond}\), and \({\it James Bond}\ \xrightarrow {{\bf Portrayer}}\ {\it Daniel Craig}\). In the example, there could be several other paths connecting the input entities (e.g., \({\it Roger Moore}\ \xrightarrow {{\bf Nationality}}\ {\it British}\), and \({\it Daniel Craig}\ \xrightarrow {{\bf Nationality}}\ {\it British}\)) but they are not relevant to the context and hence not included in the CPR result. In some cases, a query may have no paths as results. It could be that the knowledge graph does not have a path between the two query entities. Second, there may be path(s) but none is relevant to the query context. While the former can be easily handled, the latter requires relevant paths to be accurately determined.

For CPR to be interesting, the given knowledge graph should be rich in its coverage of entities and relations. This ensures that relevant paths, if exist, can be retrieved. Examples of rich knowledge graphs covering general domains include DBpedia and WordNet. There are also knowledge graphs covering specialized domains, e.g., Global Research Identifier Database (GRID) for educational and research entities.¹ In domains where knowledge graphs may not exist or may be incomplete, researchers have looked into automated construction of knowledge graphs from domain-relevant text [14, 29] or pre-trained embedding models [56].

CPR is related to knowledge graph-based explainable recommendation, which aims to discover user preference through investigating the underlying knowledge graphs constructed for items and users [2]. For instance, KGAT learns the importance of paths based on their abilities to predict the target items [45]. Another similar IR task is explainable question-answering, which extracts answer to a question by traversing the knowledge graph. CPR differs from the two by returning paths as results instead of items or entities. CPR is also different due to the consideration of query context. This, therefore, rules out the possibility of directly applying the solutions of the two tasks.

There are several technical challenges to be addressed in CPR. The first challenge is (a) the incorporation of query context in the query entity representation. Query context representation is non-trivial, because we want to capture sufficient and accurate semantics of the query context, which contains limited amount of textual content and possibly other entity mentions. The second challenge is (b) the representation of paths in knowledge graph in an inductive manner. The encoding of path should be inductive, as we not only have to encode paths in training phase, but we also need to handle new paths at query phase. The final challenge is (c) matching paths against the query entities even when they are in heterogeneous forms. Paths are formed by possibly multiple entities and relations from a knowledge graph while the query context is textual. The comparison between the two is non-trivial unless (i) they can be represented in the same space where some similarity measure in this space can be proposed to rank them in an unsupervised manner; or (ii) they are represented in different spaces and a separate model is trained to determine the relevance of path with respect to query context. While option (i) provides easier comparison between paths and contexts, it is difficult to jointly embed them in the same vector space. Option (ii), however, is more flexible, as we can train the path and context representations separately. In this article, we want to mimic the way an intelligent human user would handle the CPR task. We formulate a CPR framework based on learning embeddings for context representation and path representation, which in turn can be matched for returning the correct contextual path(s) using a supervised learning approach. For example, in the context of a movie awards ceremony news article, it is more semantically appropriate to relate an actor with a director through a movie production, than an award given out by the director to the actor at a ceremony. The CPR task has to consider both the context underlying the query entities and the path’s semantics to determine the contextual relevance of the latter for result generation.

As the number of annotated paths is expected to be small, we want our models to be able to acquire the underlying semantics of paths by introducing path representations based on the knowledge graph representations of entities and relations. We also need to capture the semantics of the given context using a good representation for matching with path representation.

Finally, this research requires datasets with ground truth paths and their associated context documents. This dataset construction requires annotators with good domain knowledge and text comprehension abilities. We therefore have to design smaller annotation steps and support them with user-friendly annotation UI.

Our contribution can be summarized as follows:

CPR is related to prediction tasks in knowledge graphs that recover missing relations and relation labels. Knowledge graph completion and relation/link prediction are two such tasks.

One related IR task to CPR is entity relation explanation. As opposed to CPR, which retrieves contextual paths given a textual input, entity relation explanation finds passages that explain a relation between two entities in a knowledge graph [1, 4, 34, 42, 43]. Entity support passage retrieval, however, aims to find a support passage that explains why an entity is connected to a given query [5, 12, 23]. In the following, we discuss in detail about two tasks that also try to retrieve knowledge graph paths to explain the relation between two entities, namely, recommendation with knowledge graph and question-answering with knowledge graph.

An important part in the definition of CPR is context. Context is an input to the framework to identify which path in knowledge graph best represents the relationship between two query entities. In our proposed framework, we look for ways to encode the unstructured context to be a vector representation. Specifically, we prefer unsupervised learning methods, as the encoding of context can be separated from other components in the framework. Classic unsupervised ways to represent a document include one-hot encoding or TF-IDF vectorization. However, such methods often suffer from sparsity. Dense word embeddings (word embeddings) research is therefore proposed to tackle the sparsity problem. These works learn the word/document representation by looking into statistics of a word in its context such as GloVe, Word2vec, and doc2vec [25, 30, 33].

State-of-the-art unsupervised text embedding works incorporate contextual information by introducing a modified transformer structure [41]. The most representative work in this field is BERT [16]. BERT utilizes multiple layers of transformer encoders that are trained to predict masked words with large text corpora so a different representation of a word is contextualized, i.e., the same word has different representations when occurring in different textual context.

While the aforementioned methods have been already proven useful in tasks such as document classification, they can only cover words. As the context may consists of both mentions of entities and words, we need to explore entity embeddings. Given an observed relation triplet \((e_h, r, e_T)\) in the KG, Translation-based entity embeddings such as TransE, TransR, and TransH learn entity and relation embeddings \(z_*\) in an alignment that satisfies the translation operation \(z_{e_h} + z_r \rightarrow z_{e_t}\) [7, 27, 47]. However, multiplication-based methods such as DISTMult and ComplEx represent a relation as a matrix \(W_r\) that satisfy \(z_{e_h}^TW_rz_{e_t}=1\) [40, 54].

Finally, we review hybrid embedding methods that jointly learn the word and entity representations in a common vector space. Wikipedia2vec jointly learns the representations of entities and words in a common vector space using skip-gram model [53]. There are also works that try to learn contextualized hybrid entity embedding based on BERT. KG-BERT augments BERT with knowledge graph by tuning a pre-trained BERT with triplet identification or relation prediction loss function [56]. Other works such as K-BERT and KEPLER further learn the entity to word interaction in a knowledge graph and text documents mentioning its entities [28, 44]. K-BERT injects relation triplets extracted from the knowledge graph into for a BERT-transformer to encode, for a BERT-transformer to encode words and entities simultaneously. Last but not least, KEPLER jointly optimizes knowledge graph loss (e.g., TransE) and masked language model loss to implicitly incorporate knowledge into contextualized word embeddings. The entity is represented by encoding the description from its Wikipedia page in KEPLER.

We formally define a knowledge graph \(G\) to be a tuple \((E,L,R,F)\) where \(E\) denotes a set of entities, \(L \subseteq E \times E\) denotes a set of edges, \(R\) denotes a set of relation labels, and for each \(l = (e_h,e_t) \in L\), \(F(l) \rightarrow R\), or simply \(e_h \xrightarrow {r} e_t\) where \(r=F(l)\). For the purpose of establishing connections between entities, we assume that a knowledge graph has both a set of relations (e.g., participateIn) and their corresponding reverse relations (e.g., participateIn\(^{-1}\)). For example, in DBpedia, John Cena and Heath Slater are two person entities connected to each other via a wrestling match denoted by \(x\), i.e., \(e_{John Cena} \xrightarrow {participatesIn} e_{x}\), and \(e_{x} \xrightarrow {{\it participatesIn}^{-1}} e_{Heath Slater}\).

Among the paths between two entities in the knowledge graph, only a few carry useful contextual semantics that can best explain the connection between the entities. We call these the contextual paths.

In the above definition, we assume that the entities \(e_h\) and \(e_t\) as well as other entities/relations in the path are not only mentioned in \(d\), but also found in the knowledge graph \(G\). This assumption is reasonable given the existence of several large text corpora of documents with entity mentions linked to knowledge graphs, e.g., Wikipedia, Freebase [6], and AIDA CoNLL-YAGO Dataset [20], and the increasingly more accurate NER and entity link methods capable of linking entity mentioned in documents to knowledge graphs. We use \(|p|\) to denote the length of a path \(p\).

Next, we formulate the Contextual Retrieval Problem (CPR) as follows:

For example, in Figure 1, Alfonso Cuarón and Children of Men are two entities in a context document containing several sentences. The CPR problem is to retrieve the contextual path(s) from knowledge graph that explains the connection between Alfonso Cuarón and Children of Men. In the figure, the path \(\langle Alfonso\ Cuar\acute{o}n, {\bf director}, Children\ of\ Men \rangle\) is a candidate contextual path.

We divide the CPR task into two subtasks: (a) construction of candidate paths connecting \(e_h\) and \(e_t\) and (b) determination of contextual paths among the candidates. For subtask (a), we can focus on a set of simple yet effective heuristics to constrain the set of candidate paths. In this article, we focus on subtask (b), which requires a new approach to context and path representations as well as incorporating them into path ranking. We shall elaborate on our solution framework below.

Our proposed the Embedding-based Contextual Path Retrieval (ECPR) framework, as shown in Figure 2, involves a knowledge graph \(G=(E,L,R,F)\) covering entities and relations of some domain. We divide the framework into two phases: training phase and query phase. During the training phase, ECPR assumes that the training data includes a set of documents \(D\) of the same domain, a set of query entity pairs \(Q\), and their corresponding contextual paths (positive paths) \(P^+\) and other non-contextual paths (negative paths) \(P^-\). \(Q\) is a set of head-tail entity pairs and the corresponding context documents, i.e., \(Q=\lbrace q_m | 1 \le m \le |Q| \rbrace\). Each query \(q_m\) is a triplet \(\langle e_h, e_t, d \rangle\), \(e_h, e_t \in E\), and \(d \in D\). Both head and tail entities \(e_h\) and \(e_t\) are mentioned in the document \(d\). Every query \(q_m\) is associated with a set of candidate paths \(P_{m}\) (size of the set is denoted by \(|P_{m}|\)). The contextual path and the set of non-contextual paths between \(e_h\) and \(e_t\) in \(d\) are denoted by \(P^+_{m}\) and \(P^-_{m}\), respectively.

In the training phase, three major components are trained, namely, context encoder, path encoder, and path ranker. First, the context encoder is trained to return context representations \(z^{cx, m}_h\) and \(z^{cx, m}_t\) of entities \(e_h\) and \(e_t\), respectively, for each query \(q_m=\langle e_h, e_t, d \rangle \in Q\). The context encoder may combine the embeddings of other words and/or entities in the relevant section of the context document as it generates the two context representations.

The second component path encoder returns a path representation for each path found in \(G\). In particular, it generates the representations \(\lbrace z^{pt,m}_{l}\rbrace\) for the candidate paths \(\lbrace p^m_{i}\rbrace\) of the query \(q_m\). While paths are sequences of entities and relations, path encoder turns them into vector representation forms that can be matched against representation of the query context. During the training phase, both context encoder and path encoder are learned from the queries with input entity pairs and their labelled contextual and non-contextual paths.

The final component is path ranker, which ranks a set of candidate paths against the query context during the query time using their representations (i.e., context representation and path representation), respectively. As each query \(q_m\) is associated with a positive candidate path \(P^+_{m}\) and negative candidate paths \(P^-_m\) in Figure 2, we assume \(p^{k}_{m}\) is the ground truth contextual path Path ranker is thus trained to differentiate the positive candidate path from the negative ones through a supervised learning model. In this article, we train path ranker using both a binary classification and a learning to rank method.

In the query phase, when given a query \(q_{m^{\prime }} = \lt e_{h^{\prime }}, e_{t^{\prime }}, d^{\prime }\gt\) consisting of entity pair \((e_{h^{\prime }}, e_{t^{\prime }})\) from the context document \(d^{\prime }\), we use the trained context encoder and path encoder (context encoding and path encoding) to obtain the context representation \(z^{cx,m^{\prime }}_{h^{\prime }}\)/\(z^{cx,m^{\prime }}_{t^{\prime }}\) as well as candidate path representations \(z^{pt,m^{\prime }}_{*}\). The path ranker will then compare each entity-path pair and rank the candidate paths by the likelihood of being the contextual path. In this work, we propose different options for context encoders and path rankers. We elaborate on the details of the three components in the framework in Sections 4, 5, and 6.

TF-IDF determines the relevance of a word or an entity to a context in document \(d\) by combining term frequency and inverse document frequency together. The TF-IDF context representation of \(d\) is thus a TF-IDF vector where each element is the TF-IDF score of a word in a vocabulary of words and entities. We learn a TF-IDF vectorizer using our dataset. The vocabulary consists of all words in entity surfaces plus the top 300 words ranked by TF-IDF found in the context document set \(D\). The size of vocabulary is chosen based on grid search on settings that yield the best MRR. Further, stop-words are removed from all context documents. We subsequently use \(z^{cx,m}_{h}(z^{cx,m}_{t})\) to represent the TF-IDF vector of \(e_h(e_t)\)’s context.

TF-IDF as a simple method suffers from sparsity of words in the context. Thus, we propose to use dense embedding methods to address the sparsity issue. Specifically, we encode the context using the averaged embeddings of tokens in the context. In this work, we choose three different average embeddings methods that cover only entities, cover only words, or cover both entities and words.

By averaging the embeddings of words and/or entities in the context, the averaged embeddings methods treat everything in the context equally instead of differentiating the entities or words by their relevance or importance to the context. For instance, for the query (Daniel Craig, Casino Royale) and the context “British actor Daniel Craig has been confirmed... The next Bond film Casino Royale is due to film in Italy, the Bahamas, the Czech Republic and Pinewood Studios,” all the location names are irrelevant to the retrieval of contextual path Daniel Craig \(\xleftarrow {{\bf starring}}\) Casino Royale. These irrelevant information should be treated as noise that should be excluded from the context representation.

Therefore, we propose the context-fused entity embedding method using a retrofitting approach and its variant(s) to incorporate background knowledge as constraints into context representations [17]. Retrofitting is originally designed to refine pre-trained word representations with synonym information from semantic lexicons and assign synonymous words to have similar representations. Counter-fitting, another form of retrofitting, repels representations of antonym words from each other [31]. At the beginning of retrofitting (counter-fitting), synonymous word pairs (e.g., good and nice) and antonymous ones (e.g., good and bad) are extracted from a thesaurus and used as constraints. The model then updates a pre-trained word embedding model to satisfy all these constraints while preserving the structure of the original embeddings. The retrofitted (counter-fitted) word embeddings not only carry all its own original attributes and semantic alignment, but also the semantics in synonymous and antonym constraints.

In context-fused entity embeddings, we use retrofitting and counterfitting to augment the original word and entity embeddings with context constraints. In other words, given the head(tail) entity embedding \(z^{KG}_{h}(z^{KG}_{t})\) from a knowledge graph embedding such as TransE, we will retrofit it with constraints made up of an entity set \(E^*\) extracted from the context. The resultant entity embedding \(z^{cx,m}_{h}(z^{cx,m}_{t})\) will then be used as context representation. For the rest of this section, we overload the notation for simplicity and represent the entity embedding of an entity \(e\) as \(z_e\). The retrofitted representation of \(z_e\) is denoted by \(z^{\prime }_e\).

This method is similar to Averaged Embedding method except for the use of contextualized representations for context encoding. Contextualized embeddings are proved to perform better than traditional embedding models (e.g., Word2vec) in tasks such as document classification [16].

Our first path ranker is effectively a binary classifier that outputs the probability of a path being a contextual path. We assume that only one path will be the ground truth for each entity pair in a document. The path ranker is trained on a set of data triples \(Q=\lbrace (q_m\), \(p_k\), \(y_{m,k}) \rbrace\) where \(q_m = (e_h,e_t,d)\), \(p_k\) and \(y_{m,k} (1:\) relevant, \(0:\) irrelevant\()\) denote the query, candidate path, and class label, respectively. With query context and candidate path encoded as \(z^{cx,m}_{h}\) and \(z^{cx,m}_{t}\), respectively, a data triple corresponds to the concatenation, i.e., \(x_{m,k} = [z^{cx,m},z^{pt,m}_k]\) where \(z^{cx,m} = [z^{cx,m}_{h},z^{cx,m}_{t}]\). For each data triple \((q_m,p_k,y_{m,k}=1) \in Q\), we randomly sample at most five negative paths from its candidate paths. We learn a classifier \(f^{BI}\) such that \(\hat{y} = f^{BI}(x)\), and simply use binary cross entropy as the loss functionwhere \(y_{m,k}\) are the ground truth labels of the candidate paths \(P^+_{m} \cup P^-_{m}\) for the \((e_h,e_t)\) pair, and \(\hat{y}_{m,k}\) are the prediction labels of the same candidate paths returned by the path ranker. We build the binary classifier as an two-layer inference neural networks in our experiments with 128-dimensional hidden layers.

While binary classifier provides a good method to rank the candidate paths, it only learns to identify the most plausible contextual path. It ignores the fact that learning to rank is a prediction task on list of options. Thus, our second path ranker, is a listwise ranker that aims to recover the ground truth ranking. The ranker is trained on a set of data triples \(Q=\lbrace (q_m\), \(p_k\), \(s_{m,k}) \rbrace ,\) where \(q_m\) and \(p_k\) are the queries and candidate path, respectively. The element \(s_{m,k}\) refers to the similarity between the ground truth contextual path and the candidate path \(p_k\), i.e., \(s_{m,k} = Cosine Similarity(z^{pt,m}_{k}, z^{pt,m}_{GT})\) for path \(p_k\). We use \(P_m\) to denote the set of candidate paths (including the ground truth contextual path) for query \(q_m\). The candidate paths are then ranked by their predicted scores \(s^{\prime }_{m,k}\)’s. The path ranker then learns to minimize the difference between its predicted rank by \(s^{\prime }_{m,k}\) values and ground truth rank by \(s_{m,k}\) values. Each query consists of \(|P_{m}|\) context representation is.

Popular learning to rank methods include LAMBDAMART [9] and listNET [11]. In this article, we utilize \(\text{XE}_{NDCG}\)MART [8] as our learning to rank path ranker. \(\text{XE}_{NDCG}\)MART learns a scoring function such that \(f^{LTR}: X \rightarrow \mathbb {R}\) where \(X\) is the set of input data. Specifically, the scoring a candidate path \(p_k\) for the query \((e_h,e_t)\) in \(d\) can be represented as \(f^{LTR}(x_{m,k}) = s^{\prime }_{m,k}\) where \(x_{m,k} = [z^{cx,m},z^{pt,m}_{k}]\). The loss function consists of a score distribution \(\rho\) and a parameterized class of label distribution \(\phi\):where \(\gamma\) is a bounding parameter and \(\gamma \in [0,1]^{|P_{m}|}\). The loss function is then the cross-entropy between the two distribution of all instances in the training set:

We use the LightGBM package to implement this LTR ranker [24].³

In the training of ECPR framework, we could either jointly optimize path encoder and path ranker together, i.e., \(\mathcal {L} = \lambda _2\mathcal {L_{\text{pt}}} + \lambda _3\mathcal {L_{\text{pr}}}\), \(\mathcal {L_{\text{pr}}} \in \lbrace \mathcal {L_{\text{BI}}}, \mathcal {L_{\text{LTR}}} \rbrace\). This way, the error of path ranker will affect the alignment of path representations. It is, however, more complex to train the model unless a large set of training data is available. We optimize the context encoder, path encoder, and path ranker separately. While the training process involves little training time, the trained model may not guarantee good performance. We leave the discussion of joint optimization to future works.

To construct a real dataset, we crowd-sourced annotations of contextual paths for two sets of Wikinews articles. Each Wikinews article serves as a context document. Wikinews is ideal for a number of reasons: (1) Wikinews articles are well written; (2) they are already classified into categories according to its topic; (3) they are Wikified, that is, the entity mentions are linked to the Wikipedia entries; and (4) the knowledge graph of entities and relations in Wikinews can be found in DBpedia. In this work, we select 40 articles under Film category and another 40 under Music category. We name the two datasets Wiki-film and Wiki-music. The Wikinews archive is publicly available.⁶ Since not every AMT worker is familiar with films, the annotation of contextual paths in Wikinews article is non-trivial. Furthermore, an entity pair could have a lot of candidate contextual paths between them. We thus split the annotation into two phases: (P1) one-hop path annotation and (P2) multi-hop path annotation. To derive longer-hop paths for annotation, we also augment the Wikinews articles with additional sentences covering more entities between P1 and P2.

While real datasets are useful for performance evaluation in real-world applications, they are costly to construct and hence too small to evaluate models with different task settings (e.g., queries with different number of candidate paths, queries with different length of contextual path...). This motivates us to construct a synthetic dataset with controllable dataset characteristics.

We compare the ECPR-based models with different component settings. Other than using PathVAE for path encoding, we experiment with two path ranker configurations, namely, binary classification and learning-to-rank methods. We also include four context encoder configurations mentioned in Section 4, namely:

For all context encoders except for the context-fused entity encoder, we consider both context window (CW) and whole document (WD) context range options for context encoding. We experimented with different context window size settings and empirically set it to be 15, as it yields good results. Recall that in Section 4.3.1 the constraints are constructed using both context window and the whole document.

To further compare the ECPR-based models with simpler retrieval methods, we include two simple baselines that do not follow the ECPR framework. Both of them do not use embedding-based path encoding. We elaborate on them as follows:

Finally, we include a random guess baseline that randomly selects a candidate path as prediction and a shortest path baseline that always predicts the shortest path (randomly choose one when there are multiple shortest paths). The performances of these two baselines serve as lower bound references for the others. We report the performance of the six types of context encoders combined with path encoder and the two path rankers (i.e., binary classifier and LTR ranker).

To obtain base entity and relation embeddings using TransE in both path and context encoders, We fix the embedding dimension size \(\mathsf {d}^{\text{bg}}=64\), which has also been used in previous works [7, 27]. We train the model for 500 epochs with early stopping. Adam optimizer is used with initial learning rate of \(10^{-3}\). All deep network structures are constructed using Pytorch.⁹

During the query phase, each ECPR-based model returns the probability of every candidate path being the contextual path for an query entity pair and context. We then rank the candidate paths by probability in decreasing order and report the Mean Reciprocal Rank (MRR) and hit@k. Both MRR and hit@k give high (or low) performance score when the ground truth paths are ranked at the top (or bottom) or near the top (or bottom). Instead of focusing solely on ground truth path retrieval, we also want to measure performance based on how similar the top-ranked candidate path(s) is similar to the ground truth path. We therefore introduce two path difference measures, NGEO and PED, to compare how the highest ranked path is similar to the ground truth path. As NGEO and PED are error-based measures, the smaller they are, the better the model performs.

Before we present other results, we first present results of path encoding with PathVAE. In Figure 4, we show the t-SNE visualization of PathVAE embeddings of a random sample of paths including four selected paths. These four selected paths share the same head entity Francis Coppola:

We want to examine if PathVAE demonstrates the property of placing similar paths close to one another and different paths from from one another. This way, one can determine if clusters form among paths. Furthermore, we want to evaluate if PathVAE can effectively handle new paths.

Among the selected paths, paths A, B, and C are similar to one another, as they share the same relation label, director. We observe their mutual closeness by PathVAE in Figure 4. Path D, however, is far from the rest, as it describes a different relation. Figure 4 also depicts paths A, B, and C in the same cluster (colored blue) while D is in another (colored orange) when we cluster the paths using K-means. This empirically illustrates that PathVAE can encode paths effectively.

Next, we check if distance between paths reflects similarity. Among paths A, B, and C, the first two are more similar to each other, as they share related tail entities, i.e., (The Godfather Part II is a sequel to The Godfather). Path C’s tail entity (Apocalypse Now) is less related to that of A and B. The locations of paths A, B, and C in Figure 4 match the above judgments. The distance between embeddings of paths A and C is larger than that between A and B. This indicates that our trained pathVAE embedding model captures path similarity well.

In addition to paths A–D, in the figure, we also show path (E) Francis Coppola \(\xleftarrow {{\bf director}}\) The Godfather(film series), which is a randomly sampled neighbor surrounding paths A–D. The visualization displays the embeddings of path E within the same cluster as paths A to C. Furthermore, as the entity The Godfather(film series) is related to The Godfather and The Godfather Part II, E is very near paths A and B in the PathVAE embedding space.

Finally, to demonstrate PathVAE’s ability to induce new paths, we show a path that does not exist in the training path set: (F) Francis Coppola \(\xleftarrow {{\bf director}}\) The Godfather Saga. Since The Godfather Saga is a television miniseries that combines The Godfather and The Godfather Part II into one film, PathVAE has correctly placed path F near paths A and B, as shown in Figure 4.

We evaluate the CPR results of ECPR-based models using different AVG embedding context encoders while using PathVAE for path encoding and learning-to-rank for path ranking. For simplicity, we assume context window (CW) to be the default context range option. As shown in Table 3, entity-word embeddings (i.e., Wikipedia2vec) performs the best among all AVG embedding encoders. Entity-only (TransE) and Word-only (Word2vec) AVG embedding options share similar poor performance. The same result is observed for both Wiki-film and Wiki-music datasets. It shows that by combining both entity and word embeddings, Wikipedia2vec can more effectively capture the context semantics than TransE and Word2vec. When there is no other entity mentioned in the context window, entity-only AVG embedding option will reduce to random guess. This may explain why entity-only AVG embedding performs slightly poorer than word-only AVG embedding. As AVG embedding using Wikipedia2vec yields the best performance, we will leave out TransE and Word2Vec options in the subsequent experiment results and discussions.

Table 4 shows the CPR results of ECPR-based models using context-fused entity embeddings, models that use constraints for retrofitting, i.e., Retrofit (I+O), outperforms Retrofit (I), and No Retrofit. The inclusion of both in-context and out-context constraints helps to augment the query entity representations with knowledge that are relevant to the context. Table 4 shows that the No Retrofit option yields the worst performance. This is reasonable, as the input in this setting is basically \([z_{e_i}, z_{e_j}]\), which do not provide any additional information and will result in performance similar to random guess. The Retrofit (I+O) option yields the best performance followed by the Retrofit (I) option. Henceforth, we will use Retrofit (I+O) as the representative context-fused entity encoding option in the subsequent experiment results.

Finally, we compare ECPR-based models that utilize contextualized embeddings. As shown in Table 5, ECPR-KEPLER and ECPR-BERT are the best- and worst-performing models. ECPR-Cxt Emb(K-BERT) is the second-best-performing model, followed by ECPR-Cxt Emb (KG-BERT). Both KEPLER and K-BERT incorporate descriptive knowledge of entities in addition to relation structure of knowledge graph during training. KG-BERT, in contrast, incorporates only relation structure of knowledge graph. This may explain the observed performance differences. Henceforth, we shall use KEPLER as the representative contextualized word-entity embeddings in subsequent experiments.

In addition to different context encoders, we compare the path rankers: binary classifier and learning-to-rank (LTR) model. We evaluate the two with ECPR-Cxt Emb using KEPLER context encoder and PathVAE path encoder. As shown in Table 6, we found that LTR path ranker outperforms binary classifier ranker. This finding is more significant in the NGEO and PED results. LTR’s list-wise ranking mechanism is more superior than binary classifier, which only has been optimized to predict the ground truth contextual path. Hence, it is appropriate to use LTR as default.

We show the result of baselines and the representative context encoders in Tables 7 and 8 and the results using the whole document option for context encoding. For MRR and hit@k, the best-performing model is ECPR-Cxt Emb with KEPLER using context window as context encoder, PathVAE and LTR ranker as path encoder and path ranker, respectively. For both datasets, ECPR-Cxt Emb outperforms ECPR-Cxt-fused followed by others. ECPR-AVG Emb and ECPR-TF-IDF share similar performance and outperform the baseline-AVG Emb and Baseline-IF-IDF by a small margin. The baseline TF-IDF with cosine similarity path ranking performs so poorly that its MRR is only 0.003 better than random guess. Generally, the two no-PathVAE baselines (TF-IDF and AVG Emb.) also perform poorly when learning to rank is used.

In summary, we conclude that ECPR-Cxt Emb > ECPR-Cxt-fused > {ECPR-AVG Emb, ECPR-TF-IDF} where > denotes “outperforms.” This suggests that context encoders that embed more information perform better. Although ECPR-AVG Emb already considers words and entities in the query context, they do not differentiate the importance of words and entities to the query context. ECPR-Cxt-fused with Retrofit (I+O), however, treats entities in and outside context window differently. It can therefore learn to exclude negative context from the entity representation by using NCR constraints and achieve better performance results. Finally, ECPR-Cxt Emb encodes context in a way that the more important information weighs more in the context representation. It distinguishes every token no matter if it is an entity or a word and is able to represent the context use both background knowledge from the pre-trained model and contextual information from the context document. Therefore, it is not a surprise for ECPR-Cxt Emb to outperform the others. Although the gaps in performance seem small, we conduct significance test on the results and concluded significant difference (p-value\(\lt 0.01\)) between or best and runner-up models.

We also compare models when using context window and whole document as context range (except ECPR-Cxt-fused). The two tables show that those using context window outperform those using the whole context document. This suggests that the whole document content dilutes the focus on query entities. It is therefore better to derive context encoding using words and entities nearby the query entities.

Next, we examine the differences between the traditional performance metrics, MRR and hit@k, with our proposed path similarity-based metrics, NGEO and PED. We observe that the similarity-based metrics generally capture performance differences more clearly. Furthermore, NGEO reflects how much modification should be made to a path to be converted to the ground truth path, and PED indicates the similarity of paths in a embedding space. This result implies that when a good model (e.g., ECPR-Cxt-Emb using KEPLER) does not rank the ground truth path at the top, it would still predict a path that is similar. We will elaborate on this in our case studies in Section 9.8.

There might be concerns about whether our models favor shorter paths over longer paths. While we find our models favor shorter paths, the averaged length of paths selected by the model for Wiki-film dataset (=3.842) are longer than that of shortest candidate paths (=2.34). We have the same observation for the averaged length of paths selected by the AMT human annotators (=3.837).

In ECPR models, we use pre-trained context encoders and path encoders. In both training and querying, context and path encoding incur very little time (\(\lt \!\!1\) ms per context/path). Thus, we spent most of the time on candidate path extraction and learning of path ranker.

Candidate path extraction could cost a lot of time, as one might need to conduct random walk on every possible entity that could be on the path from head to tail entity. To improve efficiency when extracting candidate paths, for an head entity \(e_h\) we keep a dictionary of list of entities it can reach in 1–6 steps. Before generating candidate paths between \(e_h\) and a tail entity \(e_t\) with random walk, we eliminate every \(i\)-hop neighbor of \(e_h\) if it cannot reach \(e_t\) in \(L_{MAX}-i\) steps. By doing so, we significantly reduce the time spent in candidate path extraction. On average, it takes 3.42 seconds to extract all candidate paths given a pair of head and tail entities in the query phase.

To learn a binary classifier path ranker, we spent 412 and 339 seconds for Wiki-film and Wiki-music dataset. In the query phase, it only takes \(\lt 1\) ms to provide prediction to a query. Compared to binary classifier path ranker, LTR path rankers take more time, as LTR involves a more complicated optimization process. Still, it only takes 14.3 and 11.2 minutes to train LTR path rankers for Wiki-film and Wiki-music datasets, and around 30 ms to give prediction in the query phase.

Here, we illustrate the model differences using a few case examples. In the examples, entity mentions are underlined. Mentions of query entities are in bold and underlined.

As Wiki-film and Wiki-music datasets are relatively small-sized, we experimented selected ECPR models with exact same setting as described in Section 8 on the much larger Synthetic (L) dataset. As shown in Table 9, the results observed using Sythetic (L) are similar to those in Tables 7 and 8. The best-performing model is ECPR-Ext Emb with KEPLER followed by ECPR-Ext Emb with K-BERT. The results also show that all ECPR models outperform the two baselines. This result suggests that ECPR models can effectively handle large-scale datasets.

Our second research question studies how the models perform when the candidate paths are very similar. Consider the three example paths,These paths are similar, as they only differ by their intermediate entities. A query with such candidate paths is considered difficult, as it requires the semantics of query context and candidate paths to be accurately represented for the models to determine the correct contextual path.

To conduct this evaluation, we construct two sets of queries from Synthetic (S) based on the degree of similarity among the candidate paths of these queries. For each query in the synthetic dataset, we compute the pairwise PED among all its candidate paths \(PED^P\). Small \(PED^P\) suggests high path similarity. We then derive two query sets: (Query set A) consisting of 300 queries with \(PED^P \lt 0.3\), and (Query set B) consisting of another 300 queries with \(PED^P \gt 0.5\). The average number of candidate paths per query in both query sets A and B is 8. We then evaluate the model trained on Wikinews-film on the two query sets constructed using the synthetic dataset.

Based on the results in Table 10, we first verify that query set A is more difficult than query set B. Furthermore, not only does ECPR-Cxt Emb with KEPLER outperform ECPR-AVG Emb with Wikipedia2vec on both query sets, the performance gap between two query sets for ECPR-Cxt Emb is also smaller than that for ECPR-AVG Emb. This suggests that ECPR-Cxt Emb could handle query set A with accuracy similar to query set B.

Queries with more candidate paths are likely to be more challenging than those with few candidate paths. Among the queries of Synthetic (S), we construct two subsets of 50 queries each: (Query set A) has on average 12.3 candidate paths per query, and (Query set B) has an average of 3.4 candidate paths per query. We use the model trained on Wikinews-film and evaluate on the two query sets on the synthetic dataset. We report MRR and hit@1 in Table 11. Additionally, we report the performance of a random baseline where a randomly selected candidate path is returned. We show the delta in performance between the models and this random guess baseline in brackets.

The performance of query set A is much lower than that of query set B for both ECPR-Cxt-fused and ECPR-Cxt-Emb, confirming our hypothesis that queries with more candidate paths are more difficult. Moreover, while both models significantly outperform the random baseline, ECPR-Cxt Emb consistently achieves better improvement as opposed to ECPR-AVG Emb. The increment in MRR is almost similar for both query sets A and B, suggesting that ECPR-Cxt Emb’s performance in both difficult and simple tasks are very much alike.

Finally, we answer our third research question by examining how ECPR-Cxt-fused and ECPR-Cxt-Emb perform on queries with longer contextual paths. Contextual path with more hops means that more entities and relations are needed in describing the relation between head and tail entities. When the contextual path is long, we may not find the mentions of every entity in the contextual path within the context document. There may be cases where some entities in the contextual path are not even found in the context document. Thus, such queries are considered difficult tasks.

Here, we construct two query sets from Synthetic (S) each with 300 queries: (Query set A), involving ground truth paths with length \(\ge\) 5, and (Query set B), involving ground truth paths with length \(\le\) 3. In addition, we extract a subset of A (Query set A’) in which not every path entity exists in the context document. Queries in A’ are considered the most difficult tasks. When constructing the query set, we control the average number of candidate paths per query to be 8 to avoid performance being affected by number of candidate paths. The average length of candidate paths for query sets A and B are 3.63 and 3.67, respectively. The path length difference is considered small. We show the performance in Table 12. First, query set A is clearly more difficult compared to B to both ECPR-Cxt Emb and ECPR-AVG Emb. The models perform less accurately for query set A. Furthermore, the delta between performance on A’ and A is much smaller for ECPR-Cxt Emb compared to that for ECPR-AVG Emb. This observation suggests that ECPR-Cxt Emb copes with these difficult queries better.