Ensemble graph auto-encoders for clustering and link prediction

Graph attention auto-encoder (GATE)

Traditional graph autoencoders overlook the importance of reconstructed graph structure and node features. Salehi & Davulcu (2019) introduced the GATE model to address this issue. In GATE, the encoder utilizes node attributes to acquire feature representations of the original nodes and utilizes the graph’s topology by employing stacked layers to generate node embedding representations (Weng, Zhang & Dou, 2020; Salehi & Davulcu, 2019; Cheng et al., 2024). The fundamental architecture of the attention-based graph autoencoder employs multiple encoder layers, mainly because increasing the number of encoder layers enables a deeper model hierarchy, enhancing the model’s capacity to learn and yield superior node representations (Salehi & Davulcu, 2019; Xie et al., 2022a). In addition, it allows for the comprehensive utilization of graph topology data to improve node embedding representations. Each encoder layer aggregates information from neighboring nodes based on the node neighborhood to create an embedding representation of the node (Salehi & Davulcu, 2019). The model addresses the challenge of determining attention coefficients between nodes and their neighbors by incorporating an attention mechanism with shared parameters among nodes within the same layer. The specific approach is illustrated in Eq. (5):

(4) $$ei,j(k)=Sigmoid\left(v_m^{{\left(k\right)}^T}\sigma \left(w^{\left(k\right)}{H}_i^{\left(K^{\prime }\right)}\right){+v}_n^{{\left(k\right)}^T}\sigma \left(w^{\left(k\right)}{H}_j^{\left(K^{\prime }\right)}\right)\right)$$where $w^{\left(k\right)}$, $V_m^{\left(k\right)}$, and $V_n^{\left(k\right)}$ are the parameters trainable at the kth encoder layer, $H_i^{\left(k^{\mathrm{\prime }}\right)}$ is the node representation of node i reconstructed at layer k, $H_j^{\left(k^{\mathrm{\prime }}\right)}$ is the node representation of node j reconstructed at layer k, σ is the activation function, and Sigmoid is the activation function. e_i,j(k) is the attention coefficient between neighboring node j and node i in the kth decoder layer.

The decoder uses the same number of layers as the encoder. Each decoder layer requires solving the corresponding encoder layer in reverse. In other words, each decoder layer reconstructs a node’s embedded representation based on its neighbors’ representation around the node.

Random walk graph auto-encoder (RWR-GAE)

Huang, Silva & Singh (2022) reported that existing graph autoencoder models predominantly emphasize reconstruction loss and graph structure but overlook the data distribution of potential codes in the graph, leading to a low node embedding effect. They introduced the ARGA model. However, Huang & Frederking (2019) indicated that this approach is not ideal, prompting them to propose the RWR-GAE model. Algorithm 1 describes the specific process of RWR-GAE.

The random walk algorithm commences by initiating traversal from a vertex or a series of vertices within the graph. During the traversal, all nodes are traversed with a probability of p to other nodes in the graph, while with a probability of 1-p, the walk proceeds to neighboring nodes of the current node (Huang, Silva & Singh, 2022; Hoskins et al., 2020). After each walk is completed, a probability distribution is generated, reflecting the likelihood of visiting each node within the graph. This probability distribution is employed as input for the subsequent walk, and the process iterates. This probability distribution converges when specific conditions are met, culminating in a smooth probability distribution (Hoskins et al., 2020). The random walk graph auto-encoder employs random walks to acquire a sequence of traversed nodes. This sequence of nodes is subsequently input into the Skip-gram model (Huang & Frederking, 2019; Huang, Silva & Singh, 2022). The Skip-gram model, a variant of the Word2Vec model, predicts the context of a central word given its context words (Huang & Frederking, 2019; Wou et al., 2021). The Skip-gram model takes n-node features as input proceeds through a hidden layer to derive a hidden layer representation of the nodes and reduces computational complexity through optimization using down-sampling techniques. This process allows for the prediction of context nodes and the generation of a low-dimensional representation of the nodes (Wou et al., 2021).

EGSRWR-GAE

Existing graph auto-encoder models predominantly utilize the GCN as the sole encoder (Cheng et al., 2024; Kollias et al., 2022; Salha et al., 2021). Although these models are simple, they exhibit limitations in feature extraction when a single model is an encoder (Ma, Na & Wang, 2021; Lin Li & Liu, 2021; Li, Zhang & Zhang, 2022; Xie et al., 2023). Current models primarily employ shallow architectures with poorly receptive fields, and GCNs are exclusively used for transductive learning (Wu & Cheng, 2022; Li, Zhang & Zhang, 2022). As a new encoder, the EGSRWR-GAE model employs three distinct networks, GCN, GAT, and SuperGAT, effectively addressing the drawback of sole-model usage. The model structure is illustrated in Fig. 1. EGSRWR-GAE uses these three network models to acquire node representations. These learned representations are integrated through adaptive weighting to generate new node representations. Unlike traditional graph auto-encoders, which rely on a sequential approach to learn node features, EGSRWR-GAE adopts an innovative ensemble approach to learn node feature representations using three different network models. Subsequently, the learned node features are aggregated utilizing weights to reconstruct new node features. The regenerated node representations are then employed to derive the node embedding representation matrix Z. The original graph structure is reconstructed in the decoding phase using the inner product (Tang, Yang & Li, 2022; Huang & Frederking, 2019; Salehi & Davulcu, 2019; Cheng et al., 2024; Wu & Cheng, 2022). Compared to existing models, this method utilizes multiple network models and implements the ensemble concept to address the issue of low embedding caused by the reconstruction losses in current graph auto-encoder models.

The ensemble graph auto-encoder (E-GAE) is introduced to address the limitations of existing graph auto-encoder models, characterized by disregarding node representation in reconstruction losses and the inability to capture contextual information of nodes in graph data, which results in lower embeddings. The model structure is illustrated in Fig. 2. Algorithm 2 describes the specific process of E-GAE.

The proposed EGSRWR-GAE model reconstructs new node features using three networks, whereas the E-GAE model reconstitutes the node embedding matrix by integrating three types of graph auto-encoders. The E-GAE model combines three graph auto-encoders, with each taking the node’s feature matrix X and adjacency matrix A as inputs to produce node embedding matrices Z₁, Z₂, and Z₃, respectively. The node embedding matrices are then combined using adaptive weights to yield a new node embedding matrix H, as demonstrated in Eq. (5):

(5) $$H=W_1\ast Z_1+W_2\ast Z_2+W_3\ast Z_3$$where H is the new node embedding matrix, Z_i is the ith node embedding matrix, W₁, W₂, and W₃ are trainable hyperparameters, and A’ is the reconstructed adjacency matrix.

The proposed E-GAE model is integrated from three graph auto-encoders, but the possibility of integrating more graph auto-encoders is explored. Hence, additional experiments are conducted. The effects of integrating two, three, four, and five models are compared in the experiment. GATE+EGSRWR-GAE is an integration of two models, abbreviated as GE. GATE+EGSRWR-GAE+RWR-GAE+GAE is an integration of four models, abbreviated as GERG. GATE+EGSRWR-GAE+RWR+GAE(TAGconv)+GAE(ARMAconv) integrates five models, abbreviated as GERTA. GAE(TAGconv) denotes replacing the graph auto-encoder GCN layer with TAGconv. GAE(ARMAconv) denotes replacing the graph auto-encoder GCN layer with ARMAconv. The model effects are validated on the Cora and Citeseer datasets. The experimental results are listed in Table 1. Through experimentation, it is evident that integrating more models does not always produce better results. The E-GAE model changes the architecture of the traditional model and solves the disadvantages of using a single model by integrating multiple models. Unlike previous models that rely on a single model to learn the embedded representation of nodes, which has apparent drawbacks, the E-GAE model adopts an ensemble learning approach to overcome the limitations of a single weak model. Experiments demonstrated that integrating three single weak models produces a robust model with better performance. The E-GAE model utilizes three different models to learn the topological features and node information of graphs in parallel. Each model generates a node embedding matrix Z. The node embedding matrices generated by the three models are integrated through adaptive weighting, and the three parts of the loss produced by the models are fused using weighted coefficients. The calculation of the loss employs the binary cross-entropy function. Compared to models that process data sources independently, E-GAE integrates advanced features at different levels and expands the model’s receptive field, enhancing the ability to mine graph pattern features. This strengthens the learning capabilities and improves the model’s scalability.

Metrics

The same evaluation metrics as those used by Pan et al. (2019), Huang & Frederking (2019), and Wou et al. (2021) were utilized to make the experimental results more convincing. The area under the curve (AUC) and average precision (AP) metrics were employed for the link prediction task. AUC represents the probability that the predicted value of the positive sample is greater than the predicted value of the negative sample for a pair of randomly drawn positive and negative samples from the sample, and a larger value indicates better model performance (Lin Li & Liu, 2021). AP is the area below the precision-recall (PR) curve, and a larger value indicates better model performance (Lin Li & Liu, 2021). This study divided the dataset into training, validation, and test sets. The test set contains 10% of the data, and the validation set contains 5% of the data (Pan et al., 2019; Wou et al., 2021).

Baseline

The baseline models were roughly divided into two categories: traditional methods, including DeepWalk, K-means, Spectral, GraphEncoder, DNGR, RTM, RMSC, TADW, and GCN-based methods, such as GAE, ARGA, RWR-GAE, and ARWR-GE.

Among the traditional methods, DeepWalk generates a sequence of nodes by performing random walks in the network, a process that may overlook some critical nodes. K-means results are highly dependent on the initial choice of centroids and require the number of clusters to be known or estimated in advance, which is unrealistic. Spectral is a commonly used efficient embedding algorithm. GraphEncoder is prone to overfitting when dealing with small datasets, leading to poor generalization ability. DNGR is a graph embedding model that uses graph structure to generate vector representations of nodes. RTM is a model for network structure and node-level relationships. RMSC is a multi-view clustering algorithm. TADW demonstrated that DeepWalk is equivalent to matrix factorization algorithms. GAE simplifies the original graph in GCN-based methods into a low-dimensional embedding during the encoding phase, which may lead to losing some critical structural information and is very sensitive to hyperparameters. ARGA introduces an adversarial training mechanism to enhance the robustness of the model, but this increases the complexity of model training and may lead to unstable training, also consuming more computational resources. RWR-GAE utilizes directed random walks to enhance the strength of relationships between nodes, but if the RWR-GAE parameters are not correctly chosen, it can lead to information distortion or increased noise. ARWR-GE essentially fuses the ARGA model with the RWR-GAE model to address the embedding problem.

Experiment parameters

The neural processes for graph neural network model uses 32 neurons for both layers in the encoder. The two multilayer perceptron (MLP) layers use 64 and 32 neurons, respectively. The initial learning rate is α = 0.01, β = [0.9, 0.009], and dropout = 0.6. The number of iterations on the Cora and Citeseer datasets is epoch = 500, and the number of iterations on the PubMed dataset is epoch = 4000. The trade-off coefficients of the EGAE model on the Cora and Citeseer datasets are 10⁻³, with trained iterations epoch = 200, dropout = 0.6, learning rate α = 10⁻³, and the encoder consisting of a hidden layer of 256 neurons and an embedding layer of 128 neurons. The DGAE model was initially trained with a learning rate of α = 0.01, iterations epoch = 200, and dropout = 0.6 on the PubMed dataset, and the encoder consists of a hidden layer of 32 neurons and an embedding layer of 16 neurons. The settings in the corresponding papers were maintained for the other baseline models. An E-GAE model is proposed, which ensembles GATE, RWR-GAE, and EGSRWR-GAE through adaptive weights to obtain a new node embedding matrix. The number of training epochs on the Cora data is 200, learning rate le = 0.01, step size l = 50, window size w = 30, and number of walks c = 50. The parameters of the Citeseer dataset remain the same as those of the Cora dataset. The total number of training epochs is 2,000 on the PubMed dataset, learning rate le = 0.1, and step size l = 70. The relevant parameters of the specific model are listed in Table 3. The SuperGAT network model has a head count of 12, the attention type used is MX, and the sampling rate of the edges is 0.8. The graph attention autoencoder uses a multi-head attention mechanism with a headcount of 3.

Experiment results

In the experiments, the performance of the E-GAE model for link prediction was evaluated against other models on three datasets. The results of these comparisons are detailed in Table 4. Compared to the state-of-the-art NPGNN model, the E-GAE model outperformed in AP and AUC values. The E-GAE model achieved an improvement of 0.8% in AP on the Cora dataset, 1.3% in AUC on the Citeseer dataset, and 2.0% in AUC on the PubMed dataset. These improvements demonstrate the model’s advantages in the context of link prediction. The key factor contributing to this performance gain is the ensemble approach, where different models are separately trained and then combined with adaptive weights to generate a new node embedding matrix. Compared to existing models that rely on a single technique for node embedding, this approach integrates three distinct models. This circumvents the limitations of using a single model and leads to enhanced overall model performance.

Metrics

Similarly, the same evaluation metrics for the clustering task were used as in Pan et al. (2019), Huang & Frederking (2019), and Wou et al. (2021). This study employed Acc, NMI, ARI, F1, and Precision. Acc is accuracy, and NMI is the degree of similarity of the clusters, usually by comparing the clustering results with the true labels. The larger the value, the more similar the clustering results are (Lin Li & Liu, 2021). ARI represents the degree of similarity between the clustering results and the true classes (Lin Li & Liu, 2021). Precision indicates the proportion of samples identified as positive classes by the truly positive model, and F1 represents the summed average of the precision and recall (Lin Li & Liu, 2021).

Experiment results

Five key evaluation metrics were employed in the clustering task evaluation to assess the performance of the proposed E-GAE model. Although existing graph auto-encoders typically utilize GCN as an encoder and inner product decoding (Kipf & Welling, 2016; Weng, Zhang & Dou, 2020; Xie et al., 2023), the E-GAE model takes a different approach by integrating three graph auto-encoders. This approach resulted in improved performance in clustering tasks. Experiments on the three datasets, Cora, Citeseer, and PubMed, demonstrated the effectiveness of the E-GAE model, as detailed in Tables 5–7. Compared to state-of-the-art models, the model achieved significant improvements in various metrics. For instance, on the Cora dataset, the model showed a 9.4% increase in ARI, a 2.9% increase in NMI, and a 5.8% increase in Acc. On the Citeseer dataset, the model improved Acc by 3.9% and ARI by 4.0%. The model enhanced NMI by 2.7% on the PubMed dataset and ARI by 2.9%. In the experiments, the parameter dropout significantly impacts clustering tasks. Hence, comparative ablation experiments were conducted on the clustering task for the dropout parameter, which showed relatively good results when dropout = 0.6 on the Cora and Citeseer datasets. The specific information is shown in Table 8.

These results demonstrate that the E-GAE model outperforms most existing models across multiple evaluation metrics. Clustering was performed on the Cora and Citeseer datasets to provide a visual representation of the model’s effectiveness, and t-distributed stochastic neighbor embedding (t-SNE) was employed to visualize the results in two dimensions, as illustrated in Fig. 3.