LAMP: Learnable Meta-Path Guided Adversarial Contrastive Learning for Heterogeneous Graphs

Given a HIN $\mathcal{G}=(\mathcal{V},\mathcal{E},\mathcal{A},\mathcal{R})$ denoted as $G$ for short and a set of meta-path $\{\mathcal{P}\}$ with $|\{\mathcal{P}\}|=n$, we define $\{\mathcal{G}_{\mathcal{P}_{1}}\dots\mathcal{G}_{\mathcal{P}_{n}}\}$ as meta-path sub-graphs and ${\mathcal{\hat{G}_{P}}}$ denoted as $\hat{G}$ as the integrated meta-path sub-graph. We represent the encoding function with parameter $\theta$ as $f(\cdot)$ and the augmentation function with parameter $\phi$ as $t(\cdot)$. For simplicity, we denote the network schema view by $f_{\theta}(G)$ and the meta-path view as $f_{\theta}(\hat{G})$. The primary objective for contrastive learning is:

$$\arg\max_{\theta}I(f_{\theta}(G),f_{\theta}(t_{\phi}(\hat{G}))),$$

(1)

Then for the adversarial training which tries to increase the difficulty of getting agreement in contrastive learning, the objective is:

$$\arg\min_{\phi}I(f_{\theta}(G),f_{\theta}(t_{\phi}(\hat{G}))),$$

(2)

where $I(X_{1};X_{2})$ represents the mutual information between random variables $X_{1}$ and $X_{2}$. The graph $t_{\phi}(\hat{G})=(\hat{V},\hat{E})$ retains the nodes from $\hat{G}$, but its edge set is a subset of $\hat{E}$. The insight is, we are trying to make the contrast as strong as possible while the two different view still could reach an agreement, which has been proved to be a effective optimization in contrastive learning. To bridge the min-max procedure and address potential biases, we incorporate a learnable meta-path importance parameter $\gamma\in\mathbb{R}^{1\times|\mathcal{P}|}$, which shared by $f(\cdot)$ and $t(\cdot)$. Then we put all the objective together and the refined version is:

$$\arg\max_{\theta}\min_{\phi}I(f_{\theta}(G),f_{\theta,\gamma}(t_{\phi,\gamma}(\hat{G})).$$

(3)

Consistent with prior research [40, 11], we employ InfoNCE [33] to approximate $I(X_{1};X_{2})$, detailed further in Section 4.6. Regarding $\gamma$, the insight is that $\gamma$ prioritizes longer meta-paths because the most straightforward strategy for $t_{\phi}(\cdot)$ to diminish the similarity between the two views is by preserving long meta-path instances in $\hat{G}$. Conversely, during the maximization phase, shorter meta-paths become more influential. This balanced strategy empowers LAMP to harness rich high-order information while discerning the value of different meta-paths.

Given a batch of meta-path sub-graphs $\{\mathcal{G}_{\mathcal{P}_{i}}\}=\{(\mathcal{V}_{\mathcal{P}_{i}},\mathcal{E}_{\mathcal{P}_{i}})\}$ with $i=1,\cdots,|\mathcal{P}|$, we amalgamate all of them into a singular sub-graph denoted as $\hat{G}=(\hat{V},\hat{E})$, to create the meta-path view. Here $\hat{V}=\cup_{i}\mathcal{V}_{\mathcal{P}_{i}}$ and $\hat{E}=\cup_{i}\mathcal{E}_{\mathcal{P}_{i}}$. As an illustration, Figure 2(c) depicts an integrated sub-graph derived from three meta-path sub-graphs: PAP, PSP, and PAPAP. The edge $e_{12}=(0,1,1)$ emerges since it’s absent in PAP but present in both PSP and PAPAP. For every edge $(u,v)\in\hat{E}$, we assign a vector $e_{uv}=(x_{1},x_{2},\cdots,x_{|\mathcal{P}|})$ to present its semantic information, where $x_{i}$ is set to $1$ if $(u,v)\in\{\mathcal{E}_{\mathcal{P}_{i}}\}$, otherwise $x_{i}$ is set to $0$. To further capture the semantic level information, we assign a learnable vector $\gamma\in\mathbb{R}^{1\times|\mathcal{P}|}$ that quantifies the importance of each meta-path. In the message-passing phase, we utilize $\hat{e}_{uv}=\gamma\times e_{uv}$ as the edge embedding within the meta-path view. This approach ensures that overlaps between meta-path sub-graphs are mitigated, thereby curtailing redundant message passing and rendering the meta-path view more robust compared to prior methodologies. Nevertheless, this method can lead to a dense meta-path view, an aspect we address through the proposed LAMP, detailed in the subsequent section.

The dense links of the integrated sub-graph, while capturing all given meta-path sub-graphs, can pose challenges for graph contrastive learning, as sparser graphs tend to yield more favorable results [61]. To address this issue, we introduce the Learnable Meta-Path Augmentation (LMA), a adversarial training based method aimed at learning a optimized edge prunning strategy. This ensures a sparser meta-view while overcoming manually-induced biases. LMA firstly applies a random edge dropping then a learned GCN based edge prunning strategy based according to node feature and semantic information. At the first stage, random droping will effectively decrease the graph complexity since with the growth of engaged meta-path, the integrated sub-graph will be denser and eventually approximate a compeleted which is not desirable. Besides, random dropping will provdie the LMA a dynamic integrated sub-graph in each epoch so that LMA could learn a more powerful edge-cutting strategy rather than fall into a sub-optimal solution specialized for a fixed input. For learning edge cutting strategy, each edge $e\in\hat{E}$ is correlated with a Bernoulli random variable, described as $p_{e}\sim\textrm{Bernoulli}(\omega_{e})$. An edge will be present in $t_{\phi}(\hat{G})=(V,E)$ if $p_{e}=1$ and excluded otherwise. In order to cutting off edges based on not only node features, but also semantic information, the weights $\omega_{e}$ of the Bernoulli distribution are parameterized using an MLP that takes as input the concatenation of the node embeddings obtained from a $K$-layer HGNN augmenter on $\hat{G}$ and the edge type embedding $\hat{e}_{uv}$. Thus, the edge representations can be expressed as:

$$\omega_{e}=\text{MLP}\left([h_{u}^{K};h_{z}^{K};\hat{e}_{uv}]\right).$$

(4)

For a seamless end-to-end training of $t_{\phi}(\hat{G})$, the binary nature of $p_{e}$ is transformed into a continuous variable between [0,1] using the Gumbel-Max reparametrization trick [32, 21]. Specifically:

$$p_{e}=\mathrm{Sigmoid}\left(\frac{\log(\delta)-\log(1-\delta)+\omega_{e}}{\tau}\right),$$

(5)

where $\theta\sim\text{Uniform}(0,1)$. As $\tau$ converges to zero, $p_{e}$ gravitates towards binary values, ensuring the gradient remains smooth and well-defined. Notably, this kind of edge pruning, underpinned by a stochastic graph model, has also been utilized to provide parameterized explanations of GNNs [30].

To curb LMA’s tendency for aggressive edge pruning, a regularization term $\lambda_{reg}\frac{\sum_{e\in\hat{E}}\omega_{e}}{|\hat{E}|}$ is incorporated into the objective function. The hyper-parameter $\lambda_{reg}$ dictates the quantity of retained edges. Without this regulation, LMA might opt for an extreme strategy of eliminating all edges to minimize the mutual information between $G$ and $t_{\phi}(\hat{G})$, which is counterproductive. This regularization ensures an edge ratio is maintained in $t_{\phi}(\hat{G})$ to keep sufficient information for contrastive learning. The refined objective is:

$$\arg\max_{\theta}\min_{\phi}\left(I(f_{\theta}(G),f_{\theta,\gamma}(t_{\phi,\gamma}(\hat{G}))-\lambda_{reg}\frac{\sum_{e\in\hat{E}}\omega_{e}}{|\hat{E}|}\right).$$

(6)

Of note, the meta-path importance $\gamma$ offers a holistic perspective for both $f_{\phi,\gamma}(\cdot)$ and $t_{\phi,\gamma}(\cdot)$. While $t_{\phi,\gamma}(\cdot)$ strives to maximize divergence from the network schema view, it places a premium on longer meta-paths. This is because, in contrast to the meta-path view, the network schema primarily harbors single-hop information. Conversely, during the agreement maximization phase, shorter meta-paths become more salient contributors by $f_{\phi,\gamma}(\cdot)$.

In the network schema view, for a given node $i$, we initiate the process by employing a type-specific multilayer perceptron (MLP), denoted as $MLP^{\mathcal{A}(i)}$, to transform the features $x_{i}$ of node $i$ into a unified feature space. This transformation is represented as follows:

$$h_{i}^{(0)}=MLP^{\mathcal{A}(i)}(x_{i}).$$

(7)

Here, $\mathcal{A}(i)$ presents the type of node $i$. Subsequently, we incorporate one-hot encoding to represent the semantic information of various relations. This encoded information, along with the node features, is input into a unified HGNN encoder. The specifics of this unified HGNN encoder, which operates irrespective of node types while preserving edge type embeddings, will be elaborated in the following section.It is important to note that both the network schema view and the meta-path view utilize the same HGNN encoder, denoted as $f_{\theta,\gamma}$. However, a key distinction lies in the treatment of the parameter $\gamma$: it remains frozen in the network schema view, whereas gradients are enabled for $\gamma$ in the meta-path view, allowing for adaptability in encoding different types of information.

In the context of LAMP, as outlined in eq 6, it is crucial to employ a unified HGNN that can efficiently handle both the network schema view (heterogeneous graph) and the meta-path view (homogeneous graph). While an approach could involve two distinct HGNN encoders tailored for each view, such an architecture may be inappropriate for LAMP. The core concern is that distinct encoders might produce node embedding governed by entirely different parameter sets, making it extremely hard for LAMP to meaningfully minimize similarity based on topological information. Essentially, a unified HGNN encoder fosters a harmonious link between the two views, ensuring that embedding reflects inherent structural divergence rather than encoder bias.

$$\small\hat{\alpha}=\frac{\text{exp}(\text{LeakyReLU}(a^{T}[Wh_{i}\Arrowvert Wh_{j}\Arrowvert W_{r}r_{\psi}(\langle i,j\rangle)]))}{\sum_{k\in N_{i}}\text{exp}(\text{LeakyReLU}(a^{T}[Wh_{i}\Arrowvert Wh_{k}\Arrowvert W_{r}r_{\psi}(\langle i,k\rangle)]))}.$$

(8)

The core of our approach involves utilizing the network schema view $G$ and meta-path view $\hat{G}$ for the contrastive learning mechanism. Both graphs are fed into an HGNN followed by an MLP with a single hidden layer, mapping them into a space where the contrastive loss is computed:

	$\displaystyle z_{i}^{G,\textrm{proj}}$	$\displaystyle=W^{(2)}\sigma(W^{(1)}z_{i}^{G}+b^{(1)})+b^{(2)},$		(11)
	$\displaystyle z_{i}^{\hat{G},\textrm{proj}}$	$\displaystyle=W^{(2)}\sigma(W^{(1)}z_{i}^{\hat{G}}+b^{(1)})+b^{(2)},$		(12)

where $\sigma$ denotes the Leaky Relu function. The parameters are shared between the two views’ embedding.

Adopting the strategy introduced in HeCo, we generate high-quality positive and negative pairs. We introduce a connectivity vector $C_{i}(j)$, which represents the connectivity between nodes based on the number of meta-path instances connecting them.

$$C_{i}(j)=\sum_{n=1}^{|\mathcal{P}|}\mathbbm{1}(j\in N_{i}^{\mathcal{P}_{n}}),$$

(13)

where $\mathbbm{1}(\cdot)$ represents the indicator function. Following this, we establish positive and negative samples by applying a threshold to the sorted node connectivity using $T_{pos}$. The intuition here is that node pairs with higher connectivity are more likely to belong to the same class. The contrastive loss for node $i$ can be defined as follows:

$$\mathcal{L}_{i}=-\log\frac{\sum_{j\in Pos_{i}}\exp(sim(z_{i}^{G,\textrm{proj}},z_{j}^{\hat{G},\textrm{proj}})/\tau)}{\sum_{k\in Pos_{i}\cup Neg_{i}}\exp(sim(z_{i}^{G,\textrm{proj}},z_{k}^{\hat{G},\textrm{proj}})/\tau)},$$

(14)

where $sim(u,v)$ represents the cosine similarity between vectors $u$ and $v$, and $\tau$ is the temperature parameter. The final objective aggregates the contrastive losses for all nodes:

$$\mathcal{J}=\frac{1}{|V|}\sum_{i\in V}\mathcal{L}_{i}.$$

(15)

For downstream tasks, embedding from $z^{\hat{G}}$ from the meta-path view is employed. Throughout the training process, a two-step approach is implemented for each epoch. For every epoch, in the first step, parameters within LMP are frozen, and we train the HGNN by minimizing the contrastive loss. Subsequently, in the second step, HGNN parameters are frozen while LAMP is trained with the objective of maximizing the contrastive loss.

In node classification task, we leveraged learned node embeddings to train a linear classifier in a transductive setting, utilizing all available edges during training. The distribution of node labels was consistent across datasets: 24% for training, 6% for validation, and 70% for testing. Classification performance was evaluated using Macro-F1 and Micro-F1 metrics, with results reported for the test set based on optimal validation performance (Table 3). Among all baseline methods, we report the best performance with their corresponding optimal meta-path combinations For LAMP, we report the performance with combination involving all the meta-path to demonstrate the robustness. Notably, LAMP consistently surpassed other unsupervised methods and showed remarkable efficacy against supervised models, particularly in sparser datasets like IMDB and Freebase. Crucially, LAMP operates without relying on an optimal meta-path combination, setting it apart from other methodologies. We also examined LAMP’s sensitivity to meta-path combinations (Figure 1), demonstrating its superior stability and robustness, even in comparison to supervised approaches.

To examine the sensitivity of various meta-path combinations, we conducted experiments on the ACM dataset. Our focus was to observe the variations and the min-max gap in Micro-F1 scores across all possible meta-path combinations. We considered the following candidate meta-paths: "PAP", "PSP", "PTP", "PPSP", and "-PPSP", which collectively form 26 distinct meta-path combinations, as illustrated in Figure 3. It is important to note that methods like Mp2vec and DGI were excluded from these experiments, as they are incompatible with all meta-path combinations due to their inherent design limitations and their inability to achieve state-of-the-art (SOTA) performance. The results of our experiments are presented in Table 4. In these tests, LAMP demonstrated a significant outperformance over existing unsupervised methods and even surpassed some of the supervised learning methods in terms of Micro-F1 scores. Intriguingly, current state-of-the-art methods, including HeCo and Xgoal, exhibited substantial sensitivity to the choice of meta-path combinations. This finding underscores the importance of robust meta-path handling, especially in self-supervised learning contexts, and highlights the effectiveness of LAMP in addressing this challenge.

In our experimental setup, we employ the K-means clustering algorithm for the learned node embedding. For performance evaluation, we utilize standard clustering metrics: normalized mutual information (NMI) and adjusted rand index (ARI). Recognizing the potential variability introduced by K-means due to its sensitivity to initialization, we execute the clustering process across ten independent runs and present the averaged outcomes in Table 5. Notably, the IMDB dataset is excluded from this evaluation, given its multi-dimensional label structure in HGB dataset. Furthermore, direct comparisons with supervised methodologies are omitted; these models have inherent access to label information during training and are optimized based on validation metrics. Empirical results underscore that LAMP consistently exhibits superior performance across datasets, reaffirming its effectiveness in the clustering context.

This section evaluates two distinct variants: $\text{{LAMP}}_{\textrm{w.o.mp}}$ (referred to as $\text{{LAMP}}_{var1}$) and $\text{{LAMP}}_{\textrm{w.o.unifiedHGNN}}$ (referred to as $\text{{LAMP}}_{var2}$). For the $\text{{LAMP}}_{var1}$ version, we freeze the parameter $\gamma$ to cancel out the effect of meta-path importance during LMA learning. The intent behind this is to examine the role of meta-path importance in bridging local and high-order information. On the other hand, $\text{{LAMP}}_{var2}$ replaces the unified HGB encoder with the meta-path and network-schema encoders from HeCo. Within this setup, the meta-path view is processed using the HAN [46] attention mechanism, while a standard GCN tackles the original HIN. For the meta-path view, the LMA edge-pruning technique is applied to each individual meta-path sub-graph.

Table 3 illustrates that both $\text{{LAMP}}_{var1}$ and $\text{{LAMP}}_{var2}$ suffer a considerable decline in performance. (1) Lacking the meta-path importance $\gamma$, $\text{{LAMP}}_{var1}$ struggles to harness sufficient overall structural data. It primarily emphasizes local details based on node attributes. Similarly, without the guidance of meta-path importance $\gamma$, LMA tends to prioritize lengthy meta-paths, and neglect potentially valuable shorter meta-paths. The resultant effect weakens $\text{{LAMP}}_{var1}$’s capability to bridge local and high-order information. This underscores that the guidance from meta-path importance is crucial for the LAMP model. (2) For $\text{{LAMP}}_{var2}$ , employing separate HGNN encoders for the two views might have been effective in HeCo, but it does not work for LAMP. As shown in Table6, $\text{{LAMP}}_{var2}$ lags behind in performance across all datasets.Using disparate HGNN encoders inherently amplifies the differences in embedding produced by the two views, even when the target node attributes remain consistent across both views. This introduces a dilemma for LMA, making it challenging to determine which edges to prune, as the two views already appear distinct. This inconsistency can destabilize the model, increasing the risk of training collapse.

In this section, we examine our model’s sensitivity to two critical hyper-parameters: the threshold for positive samples $T_{pos}$ and the regulation term $\lambda_{reg}$, which determines the proportion of retained edges in LMA. Node classification on the ACM and DBLP datasets is evaluated, with both Macro-F1 and Micro-F1 scores presented.

HGCL has rapidly evolved, effectively adapting contrastive learning techniques for heterogeneous graphs [35, 25, 47, 24, 3, 57, 60]. Standard HGCL approaches involve creating multiple graph views via meta-path or network-schema based augmentations, followed by representation learning through contrasting positive and negative samples. DMGI [35], for instance, contrasts the original network with its corrupted counterpart for each meta-path view, integrating a consensus regularization for meta-path fusion. HeCo [47] introduces two augmentation techniques—meta-path sub-graph view and network schema view—and minimizes the inter-view information entropy using personalized pairwise InfoNCE. HDMI [25] and XGOAL [24] are advanced versions of DGMI. HDMI improved semantic attention via high-order mutual information, XGOAL proposed a stronger positive and negative samples generating strategy, and node embeddings are obtained by simply average pooling over these layer-specific embeddings. CPT-HG [23] presents a pre-training model grounded in contrastive learning by making sub-graphs derived from positive samples integrate randomly swapped nodes from the negative set.

In recent years, heterogeneous graph neural networks (HGNNs) as general extension of homogeneous graph [19, 43, 18, 45, 55, 44, 14] have risen to prominence as a pivotal tool in information retrieval (IR), adept at extracting rich structural and semantic information from heterogeneous graphs. This capability has led to their widespread application across various IR domains, including search engines, recommendation systems, and question-answering systems, among others. In the context of search engines and matching, Chen et al. [2] innovated a cross-modal retrieval method utilizing heterogeneous graph embeddings. This method adeptly preserves cross-modal information, overcoming the limitations of traditional approaches that often lose modality-specific details. Similarly, Guan et al. [12] addressed fashion compatibility modeling by integrating user preferences and attribute entities within a meta-path-guided HGNN framework. Additionally, Yuan et al. [52] introduced the Spatio-Temporal Dual Graph Attention Network (STDGAT) for intelligent query-Point of Interest (POI) matching in location-based services. By leveraging semantic representation, dual graph attention, and spatiotemporal factors, STDGAT enhances matching accuracy, even with partial query keywords.The domain of recommendation systems has also seen significant advancements through the application of HGNNs. Cai et al. [1] proposed an inductive heterogeneous graph neural network (IHGNN) model tailored for cold-start recommendation scenarios, addressing the challenge of sparse user attribute data. Pang et al. [34] developed a personalized session-based recommendation method using heterogeneous global graph neural networks (HG-GNN), which effectively captures user preferences from both current and historical sessions. Moreover, Song et al. [38] presented a self-supervised, calorie-aware heterogeneous graph network (SCHGN) for food recommendations, integrating user preferences and ingredient relationships to enhance the recommendation quality.In the arena of question-answering systems, HGNNs have garnered considerable attention. Feng et al. [7] proposed a document-entity heterogeneous graph network (DEHG) that integrates structured and unstructured information sources for multi-hop reasoning in open-domain question answering. Furthermore, Gao et al. [9] introduced HeteroQA, employing a question-aware heterogeneous graph transformer to assimilate multiple information sources from user communities, enriching the question-answering process.