Context Graph

A Triple-based Knowledge Graph (denoted as $\mathcal{KG}=\{\mathcal{E},\mathcal{R},\mathcal{T}\}$) can be represented as a set of triples in the form of $(h,r,t)\in\mathcal{T}$, where $h,t\in\mathcal{E}$, $r\in\mathcal{R}$. The notations $h$ and $t$ denote the head and the tail entity of a triple. $\mathcal{E},\mathcal{R},\mathcal{T}$ are the set of entities, relations, and triples, respectively. Typical triple-based KGs include Freebase Bollacker et al. (2008), WordNet Miller (1995) and DBPedia Lehmann et al. (2014). In these triple-based KGs, the triple representation excludes crucial contextual information, often resulting in inaccurate knowledge storage, incomplete representation, and ineffective reasoning. These issues are the primary constraints on the practical application of most current KGs.

UTF8gbsn To be specific, the same relationship may have different meanings in different contexts, thus the triple representation could lead to incorrect knowledge storage. For instance, consider the two sentences: ’A先生住在上海虹桥希尔顿酒店，闵行区红松东路’ and ’A先生住在北京市海淀区’ as shown in Figure 1(a). They may be represented as two triples: (A先生, 住在, 上海市闵行区) and (A先生, 住在, 北京市海淀区), respectively, in a KG. However, these representations are semantically conflicting since a person cannot live in two places simultaneously. This mistake is likely to occur because the predicate ’住’ in the first sentence implies ’stay in’, whereas in the latter one, it denotes ’live in’. The triple extraction process filters out the sentence context, leading to information conflicts.

Moreover, each data instance in a KG strictly adheres to its ontology structure. The ontology structure defines the categories of entities, relations, and attributes, as well as their hierarchical relationships. During the construction of a KG, knowledge outside the pre-defined categories is filtered out, including a large amount of contextual information, leading to incomplete knowledge representation. For example, the contexts of Steve Jobs serving as the chairman of Apple Inc. twice are very different as shown in Figure 1(b). However, based on triple representation, both events would be represented as (Steve Jobs, chairman of, Apple Inc.), which results in downstream tasks not obtaining sufficient information when utilizing related knowledge.

Triple-based knowledge representation also limits the effectiveness of existing KG reasoning methods, which mainly focus on learning explicit or implicit rules through rule mining or embedding models. For example, from triple (X, works in, Y) and (Y, city of, Z), it is very likely for KG reasoning models to deduce that (X, citizen of, Z) since such rule pattern appears frequently in the training data as shown in Figure 1(c). However, these probability based rules may not hold in all contexts, leading to conclusions that do not align with the facts. Besides, triple-based KGs only contain relational knowledge limited by predefined relation set $\mathcal{R}$. The triple-based reasoning process have difficulties in answering questions involving relations out of $\mathcal{R}$ without additional contextual information or external data sources.

To address the limitations of triple-based KGs, a promising approach is to attach contextual data to factual triples. For instance, several KGs, such as YAGO and the Yahoo Knowledge Graph, include meta-information with their facts, such as the time of validity, the geographic location of a fact, and provenance information. By integrating such data, CGs can offer a more comprehensive and accurate representation of knowledge, thereby enabling more effective reasoning.

As shown in Figure 2, contextual data can be roughly classified into two categories, i.e., entity contexts and triple contexts.

Entity contexts refer to information that provides a deeper understanding of an individual entity within the KG. This type of context helps in defining the attributes, characteristics and backgrounds of the entity. Entity contexts include entity attributes, entity types, entity descriptions, entity aliases, entity reference links, entity images, entity speeches, entity videos, etc.

Relation contexts refer to specific pieces of information that describe the relations between entities. They provides concrete data points and factual statements that contribute to the KG’s informational content. Relation contexts include temporal information, geographic locations, quantitative data, provenance information, confidence levels, event-specific details, and other supplementary information. By incorporating these relation contexts, KGs can offer a richer, more detailed representation of the relationships between entities, enhancing their overall accuracy and utility for reasoning and analysis.

Table 1 demonstrates some examples of different types of entity contexts and relation contexts.

A Context Graph (denoted as $\mathcal{CG}=\{\mathcal{E},\mathcal{R},\mathcal{Q},\mathcal{EC},\mathcal{RC}\}$) can be represented as a set of factual quadruples in the form of $(h,r,t,rc)\in\mathcal{Q}$, where $h,t\in\mathcal{E}$, $r\in\mathcal{R}$ and $rc\in\mathcal{RC}$. The notations $h$ and $t$ denote the head and the tail entity of a factual quadruple, $r$ denotes the relations between $h$ and $t$, and $rc$ denotes . $\mathcal{EC},\mathcal{RC}$ are the set of entity contexts and relation contexts. Each entity $e\in\mathcal{E}$ and its entity context $ec\in\mathcal{EC}$ form a complete entity representation $(e,ec)$.

To validate whether contextual information can be used to enhance the ability of KG reasoning models, in this paper, we consider two KG reasoning tasks for verification, i.e., KG completion (KGC) and KG question answering (KGQA).

Currently, commonly used KG datasets, such as FB15k237, YAGO3-10, and Wikidata5M, are encyclopedic KGs that encapsulate general knowledge about the real world. These KGs are typically developed by domain experts by applying named entity recognition and relation extraction techniques on Wikipedia. However, during this construction process, the rich contexts surrounding the entities are often omitted. Recent studies Wang et al. (2021, 2022b) have proposed to incorporate entity labels and descriptions as supplementary information for KGs. Nevertheless, the labels and descriptions are insufficient to replace the specific contexts associated with KG triples, thereby limiting their effectiveness in addressing diverse knowledge reasoning problems.

To incorporate related contexts into KGs, we consider using Wikidata and Wikipedia as our primary contextual corpus in this work. Due to the extensive coverage and up-to-date information of Wikidata, some KGs like Freebase and YAGO provides official mapping files which can map their entities to Wikidata QIDs. For entities in other KGs, we can use entity search engines provided by Wikidata to find the Wikidata entities which are most likely to be identical to searched entities. Furthermore, Wikidata provides links to the associated Wikipedia pages of its entities. Thus, we can provide contextual information from Wikidata and Wikipedia to different KGs.

In this section, we demonstrate a new context-enriched KGC method based on our proposed CGR³ paradigm. Since KGC can be considered as an entity ranking task for single-hop reasoning questions, it is not necessary to perform iterative reasoning processes. Thus, the reasoning step is omitted for this task.

In this section, we introduce an in-context learning paradigm for the KBQA task (see Figure 5). This paradigm focuses on the integration of contextual information, which plays a pivotal role in identifying plausible reasoning paths and facilitating the derivation of final answers.

Given a question $q$, we first identifies a set of $k$ topic entities $E^{(0)}=\{e_{i}^{(0)}\}_{i=1}^{k}$ with an LLM. Starting from these topic entities, we iteratively explore plausible reasoning paths until the LLM determines that it can answer the question based on the support of triples along the paths and their associated contexts. Therefore, during the inference process, we maintain and update a set of reasoning paths $P=\{p_{1},p_{2},...,p_{M}\}$ alongside a list of relation context sentences $C=\{rc_{1},rc_{2},...,rc_{N}\}$. Here, $M$ represents the width of the beam search, while $N$ denotes the number of relation context sentences. Each iteration of the process consists of three steps: 1) knowledge exploration, 2) reasoning path pruning, and 3) context-aware reasoning.

At the beginning of the $D$-th iteration, each reasoning path consists of $D-1$ triples, i.e., $p_{i}=\{(h_{n}^{(d)},r_{n}^{(d)},t_{n}^{(d)})\}_{d=1}^{D-1}$, where $h_{n}^{(1)}$ is a topic entity from $E^{(0)}$, $t_{n}^{(d)}=h_{n}^{(d+1)}$ ensures the tail entity of one triple becomes the head entity of the next. ^†††WLOG, We only look for paths with forward relations. For each triple $(h,r,t)$, we introduce a reversed relation $r^{-1}$ and the reversed triple $(t,r^{-1},h)$ into the KG.

We evaluate our proposed framework on two widely-used datasets FB15k237 Toutanova et al. (2015) and YAGO3-10 Rebele et al. (2016). FB15k237 is derived from Freebase Bollacker et al. (2008), an encyclopedic knowledge base containing general knowledge about topics such as celebrities, organizations, movies, and sports. YAGO3-10 is a subset of YAGO3 Rebele et al. (2016), a knowledge base built upon Wikipedia, WordNet Miller (1995), and GeoNames Bond and Bond (2019). To prevent potential data leakage, FB15k237 excludes reversible relations from the backend KB. Detailed statistics of the two datasets are shown in Table 2.

In this section, we evaluate the efficacy of our proposed $\text{KGR}^{3}$ framework by integrating it with three widely utilized embedding-based KGC models: RotatE Sun et al. (2019), ComplEx Trouillon et al. (2016), and GIE Cao et al. (2022). These models not only serve as baseline methods but are also foundational for candidate answer retrieval. Instead of surpassing all baseline methods, our main objective is to evaluate the effectiveness of our context-enriched $\text{KGR}^{3}$ framework when applying to different embedding models. Hence, we deliberately include a limited selection of baseline models.

We conduct all of our experiments on a Linux server with two Intel Xeon Platinum 8358 proces- sors and eight A100-SXM4-40GB GPUs. We choose the framework provided by the GIE Cao et al. (2022) project for training the base embedding models, strictly following the parameter settings provided. During the reasoning stage, we utilize OpenAI’s gpt-3.5-turbo-0125 checkpoint ^§§§https://platform.openai.com/docs/models. The Re-ranking stage employs Meta-Llama-3-8B-Instruct with BF16 precision as the backbone model ^¶¶¶https://huggingface.co/meta-llama/Meta-Llama-3-8B-Instruct. The SFT task is implemented based on the LLaMA-Factory Zheng et al. (2024) framework and applies LoRA technique Hu et al. (2021), with a rank setting to 16 and an alpha setting to 32. Additionally, AdamW Loshchilov and Hutter (2017) is used as the optimizer, the batch size is set to 2 per device, the gradient accumulation steps is set to 4, and the learning rate is 1.0e-4. The sampling ratio of the validation set is 5%, and the best checkpoint is selected based on evaluation loss.

For each query triple in the form of $(h,r,?)$ or $(?,r,t)$, the KGC model outputs a ranked list of all entities in the KG. For a fair comparison, we adopt the “filtered” setting introduced in Bordes et al. (2013). Except for the ground truth entity, we remove all other valid entities that conform to an existing triple in training, validation, or test set from the ranked list in advance. Based on the position of the ground truth entity, We compute Hits@$1$, Hits@$3$, Hits@$10$ and mean reciprocal rank (MRR), where higher results indicate better performance.

Table 3 summarizes the performance of the $\text{KGR}^{3}$ framework on three different base embedding methods. The experiment results show that $\text{KGR}^{3}$ and its simplified variants $\text{KGR}^{2}$ without “reasoning” module significantly and consistently enhances each embedding method among all metrics. On average, our $\text{KGR}^{3}$ framework improves the Hits@$1$ by 33.04% and 14.78% on FB15k-237 and YAGO3-10 datasets. These results demonstrate the effectiveness and superiority of integrating LLMs and entity contexts with embedding-based KGC models, which address our RQ1.

Notably, the improvement in Hits@1 is more substantial than that in Hits@3 and Hits@10. This indicates that the $\text{KGR}^{3}$ framework is particularly effective at identifying the most accurate answers. Since our framework primarily focuses on re-ordering top-$n$ (or top-$\delta$ if we consider reasoning outputs) entities from the initial ranked entity list, the upper bound of Hit@1, Hit@3, and Hit@10 are implicitly constrained by the Hits@$n$ or Hit@$\delta$ performance of the base embedding model. Given that Hits@1 is typically further from this upper bound, the potential for improvement will be greater. Additionally, by leveraging semantic knowledge from entity contexts, the LLM gains a more comprehensive understanding of the entities, thereby enabling more precise inferences, particularly for top-ranked candidate answers.

Furthermore, the performance gains are more pronounced for simpler embedding models such as ComplEx Trouillon et al. (2016). Simple embedding models cannot fully capture the structural information in the KG, leading to the introduction of noisy entities in the candidate answer list. With entity descriptions, the LLM can utilize its semantic understanding capabilities to identify and deprioritize candidate answers that do not match the semantics of the query triple. Hence, $\text{KGR}^{3}$ can enhance the robustness of these embedding models.

In addition, a comparison between $\text{KGR}^{2}$ and $\text{KGR}^{3}$ reveals that the inclusion of the “reasoning” provides a notable boost. In certain scenarios, the KG may lack sufficient structural information to derive plausible answers. Nevertheless, long Wikipedia paragraphs can effectively augment specific entities with extra semantic knowledge, which allows the LLM to generate additional candidate answers with its semantic reasoning capability. This surpasses the inherent limitations of KGs, leading to substantial performance improvements. An case study showing the effectiveness of the Reasoning and Re-ranking processes is demontrated in Appendix A.1-A.3

We note that a lot of commonly-used KGQA benchmarks like CWQ Talmor and Berant (2018) and WebQSP Yih et al. (2016) are constructed from Freebase Bollacker et al. (2008) which has been defunct since 2015. Some of the knowledge in Freebase is outdated or contradicts information in Wikipedia Xu et al. (2023). Clearly, compared to Freebase, the knowledge in Wikipedia has higher coverage and accuracy, and in this work, Wikipedia serves as the main source of contextual information. Our assumption is that the contextual information can support or complement the triple-based knowledge in the KG, rather than contradict it. Therefore, we consider KGQA datasets based on Wikidata where the triple-based knowledge is better aligned with the contextual information from Wikipedia, rather than KGQA datasets constructed from Freebase.

In this work, QALD10-en Usbeck et al. (2023) and WikiWebQuestion (WWQ) Xu et al. (2023) are used as KGQA datasets for evaluation. QALD10-en is a new, complex, Wikidata-based KGQA benchmarking dataset as the 10th part of the Question Answering over Linked Data (QALD) benchmark series. WWQ is constructed by migrating the popular WebQSP Yih et al. (2016) benchmark from Freebase to Wikidata, with updated SPARQL and up-to-date answers from the much larger Wikidata.

For all datasets, exact match accuracy (EM) is used as our evaluation metric following previous works (Li et al., 2023; Sun et al., 2024).

We compare with standard prompting (IO prompt) (Brown et al., 2020b), Chain-of-Thought prompting (CoT prompt) (Wei et al., 2022), and Self-Consistency (Wang et al., 2023) with 6 in-context exemplars and "step-by-step" reasoning chains. Moreover, for each dataset, we pick previous state-of-the-art (SOTA) works for comparison. We notice that fine-tuning methods trained specifically on evaluated datasets usually have an advantage by nature over methods based on prompting without training, but sacrificing the flexibility and generalization on other data. Therefore, we compare with previous SOTA among all prompting-based methods and previous SOTA among all fine-tuned (FT) methods respectively. With regard to previous prompting-based methods, we select their results achieved with GPT-3.5 for a a fair play.

We use ChatGPT (GPT-3.5-turbo) as the backbone LLM for CGR³ by calling OpenAI API. The maximum token length for the generation is set to 256. In all experiments, we set both width $M$ and depth $D_{max}$ to 3 for beam search. We use 5 shots in CGR³-reasoning prompts for all the datasets.

Since CGR³ uses external KGs and contextual information to enhance LLM, we first compare it with those methods leveraging external knowledge as well. As we can see in Table 6, even if CGR³ is a training-free prompting-based method and has natural disadvantage in comparison with those fine-tuning methods trained with data for evaluation, CGR³ still achieves new SOTA performance in both datasets. If comparing with other promoting-based methods with ChatGPT as backbone models (especially ToG), CGR³ can win the competition on all datasets.

It is noteworthy that other prompting-based methods rely solely on triple knowledge from KGs, whereas CGR³ allows the LLM to leverage additional contextual information for more precise reasoning on KGs. This is likely the primary reason why CGR³ outperforms other prompting-based methods. To verify this, we evaluated a variant of CGR³ that excludes contextual information for comparison. As shown in Table 6, incorporating contextual information results in a relative increase of 43.6% and 17.1% in Exact Match (EM) on QALD10-en and WWQ, respectively. These experimental results support our hypothesis that KGQA methods can significantly benefit from the integration of contextual information.