Unsupervised Graph Representation Learning with Cluster-aware Self-training and Refining

To alleviate the dependency on abundant manual annotations, unsupervised learning techniques that train the model with predefined pretext tasks is attracting increasing interests. The pretext tasks constructed from the raw labels can produce general embeddings for various downstream machine learning tasks of interest. Many strategies for pretext learning tasks, such as image in-painting [29], jigsaw puzzles [27, 28], grayscale image colorizing [23, 51], and geometric transformation recognition [16], have been proposed recently. However, the methods using the classification objective, which minimizes the cross-entropy loss, still obtain the best performance [2, 6]. Along this line, many methods focus on how to obtain proper labels for the classification task. For example, DeepCluster [6] is proposed to iteratively cluster images using kMeans; the cluster assignments are then fed as supervision to train the convolutional network. NAT [3] proposes to fix a set of randomly initialized target vectors and train the network by aligning the embeddings to targets. Recently, Asano et al. [2] combine representation learning and clustering and propose a novel self-labeling scheme to balance the size of clusters, which outperforms existing methods.

Representation learning on graphs is far more complex than image data due to the non-Euclidean property and the lack of spatial locality. Classical methods learn unsupervised graph representation based on random walks [4, 17, 31, 33], which sample random walk sequences from the graph and learn node embeddings using sequential models. Apart from random-walk-based methods, another line of development focuses on matrix factorization techniques [37, 44], which produce node embeddings by explicitly factorizing the proximity matrix. Notably, Qiu et al. [32] manage to theoretically unify random walks and matrix factorization techniques into a cohesive framework. Recently, to accelerate computational tasks on large-scale graphs, NRL-MF [25] proposes a network representation lightening framework based on matrix factorization. This approach employs both hashing and quantization approaches and has delivered remarkable performance in node classification and recommendation tasks. However, traditional network embedding methods may suffer from insufficient representation ability, because they generate embedding for each node independently and no parameters are shared between nodes [18].

GNNs, however, encode both structure and node features into dense embeddings via message passing in the local neighborhood and achieve strong expressive power [22, 40, 45, 46]. Nevertheless, most GNNs are established in the (semi-)supervised setting, and requires substantial label annotations to be trained. In reality, it is expensive to obtain high quality label annotations. To mitigate the problem of label scarcity, there is a growing body of literature focusing on unsupervised training of GNNs. One line of research work proposes to employ GNNs as autoencoders [21, 43], which formulates unsupervised learning as a link prediction problem and leverages the raw graph structures as supervision. The representative GAE method [21] optimizes node embeddings by reconstructing the original graph topology from learned representations. To take node attributes into consideration, Gao and Huang [14] propose to use two graph autoencoders to preserve the proximity of both graph topology and node attributes, respectively. As another promising research direction, a number of research work focuses on training unsupervised GNNs based on the InfoMax principle [26]. Pioneering work DGI [41] utilizes a contrastive objective that discriminate node embeddings from the original graph and a corrupted graph, whose training objective is proved to be a lower bound of the Mutual Information (MI) between the input graph and the learned representations. Follow-up work proposes the graphical mutual information (GMI) [30] that takes mutual information between edges into consideration. Following this work, Zhu et al. [53] propose to generate graph views with stochastic graph augmentation functions and directly maximize the agreement between node embeddings across views.

Despite their success, Tschannen et al. [38] point out that the embedding equality is not strongly correlated with the MI bound and the stricter bound can even bring worse performance. In other words, the success behind the above contrastive learning models may be attributed to the design of augmentation and contrastive architectures. Similarly, our proposed CLEAR approach also involves information maximization. However, different from maximizing the mutual information between graph representation, our approach directly optimize the information between representation and clustering assignments using the cross-entropy objective.

In the deep learning era, a number of methods solve the graph clustering problem using GNNs. Zhang et al. [52] propose to address the graph clustering problem via a plain neighborhood aggregation scheme; their proposed method simply aggregates information from neighborhood nodes without parameters to learn from data. Wang et al. [42] propose to use a deep-clustering-based method on node embeddings for graph clustering, where a cluster hardening loss [39] is introduced to emphasize the clusters with high confidence. Moreover, the same deep clustering scheme has been applied to semi-supervised learning with few labels as well [36]. In this work, the clustering algorithm incrementally generates labels for unlabeled data from labeled nodes belonging to the same cluster.

Among the aforementioned methods, none tries to directly solve the unsupervised graph representation problem. On the contrary, our work aims to learn discriminative graph representations without label supervision. By utilizing cluster information as supervision signal, which reveals an intrinsic property of the graph, our approach produces high quality embeddings that benefit a variety of downstream tasks.

Consider an input graph \(\mathcal {G} = (\mathcal {V}, \mathcal {E})\) , where \(\mathcal {V} = \lbrace v_1, v_2, \ldots , v_N \rbrace\) denotes the set of nodes and \(\mathcal {E} \subseteq \mathcal {V} \times \mathcal {V}\) denotes the set of edges. We denote \(\mathbf {X} \in \mathbb {R}^{N \times M}\) and \(\mathbf {A} \in \mathbb {R}^{N \times N}\) as the node feature matrix and the adjacency matrix, respectively, where \(\mathbf {A}_{ij} = 1\) iff \((v_i, v_j) \in \mathcal {E}\) and \(\mathbf {A}_{ij} = 0\) otherwise. The goal of unsupervised graph representation learning is to learn a low-dimensional representation \(\mathbf {h}_i \in \mathbb {R}^D\) for each node \(v_i \in \mathcal {V}\) with no access to ground-truth labels, where D is the dimension of node representations and \(D \ll M\) . We summarize all notations used throughout this article in Table 1 for better readability.

Typically, GNN models are trained using the classification objective in a supervised manner. In our unsupervised model where no ground-truth labels are given, we generate pseudo-labels to provide self-supervision by iteratively performing clustering on the embeddings. To be specific, in this unsupervised training phase, CLEAR alternates between optimizing parameters of the GNN model by predicting cluster labels and adjusting the cluster assignments of nodes.

Representation learning via graph convolutional networks. We use Graph Convolutional Networks (GCNs) [22] as the base model to learn node embeddings. GCN is a multilayer feedforward network in the graph domain that generates node embeddings by aggregating and transforming information from neighboring nodes. We define \(\mathbf {H}^{(t)}\) as the output of the tth layer. The propagation rule of each layer can be defined aswhere \(\widetilde{\mathbf {A}}\) is the normalized adjacency matrix of \(\mathbf {A}\) with self-loops added, \(\widetilde{\mathbf {D}}\) is the degree matrix for \(\widetilde{\mathbf {A}}\) with entries \(\widetilde{\mathbf {D}}_{ii} = \sum _{j=1}^N \tilde{\mathbf {A}}_{ij}\) , \(\sigma\) denotes the activation function, e.g., \(\operatorname{ReLU}(\cdot) = \max (0, \cdot)\) , and \(\mathbf {W}^{(t)}\) is the trainable weight parameter of the tth layer. The input to GCN is the feature matrix, i.e., \(\mathbf {H}^{(0)} = \mathbf {X}\) . We employ an L-layer GCN to produce node embeddings for nodes, i.e., \(\mathbf {H} = \mathbf {H}^{(L)}\) .

Self-training with cluster assignments. In CLEAR, cluster labels are used as pseudo-supervision for model training. A cluster in the graph is a group of nodes that are closely correlated to each other in terms of both topology structures and features. A variety of clustering methods have been developed, and in our article, we choose kMeans due to its simplicity. Assume that the cluster labels of \(v_i \in \mathcal {V}\) is denoted by \(y_i \in \lbrace 1, 2, \ldots , K\rbrace\) , drawn from a space of K possible clusters. We denote the cluster assignments by \(\mathbf {C} \in \lbrace 0, 1\rbrace ^{N \times K}\) , where each row represents the cluster assignments of one node using one-hot encoding. Conventional kMeans aims to learn the centroids \(\mathbf {\mu }_1, \mathbf {\mu }_2, \ldots , \mathbf {\mu }_K\) and cluster assignments \(y_1, \ldots , y_N\) by optimizingHere, we slightly abuse the notation \(\mathbf {\mu }_{y_i}\) being the centroid that has the label \(y_i\) .

To train the model without human annotations, we ask the model to predict cluster labels. To this end, we employ a MultiLayer Perception (MLP) network as the classifier. The MLP takes node embeddings \(\mathbf {H}\) as input and predicts correct labels on top of these embeddings. For a typical classification problem with deterministic labels, we solve the following optimization problem:where \(p(\mathbf {y} \mid v_i) = \operatorname{softmax} (\operatorname{MLP}(\mathbf {h}_i))\) is the prediction for node \(v_i\) .

Cluster reassignment with equipartitioning. We note that it is nontrivial to directly adopt kMeans for clustering-based self-supervised representation learning. This can be seen by a fact that when we optimize the embeddings \(\mathbf {H}\) along with cluster assignments \(\mathbf {y}\) , a trivial solution can be obtained by mapping all nodes to the same point in the embedding space and treating them as a single cluster [6].

To address this problem, we propose to adjust the cluster assignments with a novel equipartition strategy. Formally, we first treat the cluster assignments \(\mathbf {C}\) as probability distributions \(c(\mathbf {y} \mid v_i)\) . We further restrict each cluster to be equally partitioned [2]. With that requirement, Equation (3) can be rewritten as follows [2]:where \(c(y \mid v_i) = \mathbf {c}_{iy}\) is the cluster assignment for node \(v_i\) . The first requirement guarantees that each node belongs to exactly one cluster and the second ensures that all N nodes are equally split into K clusters. Then, optimizing with respect to \(c(\mathbf {y} \mid v_i)\) for all nodes is equivalent to reassigning cluster labels that satisfy the equipartition requirement.

Please kindly note that the equipartition requirement should be regarded as a regularization that aims to avoid the downgraded trivial solution, rather than a constraint that requires the input data to be in equally sized clusters. Moreover, considering we are agnostic to the number of classes in an unsupervised setting, we set the number of classes to be relatively larger to the real numbers (to which we refer as the overclustering strategy) following previous work [2, 6]. The overclustering strategy decomposes each data cluster into smaller sub-clusters and thereby allows these smaller sub-clusters to be in a similar size.

Due to the combinatorial nature of Equation (4), we resort to optimal transportation to solve it efficiently [2]. Specifically, we relax the cluster assignments \(\mathbf {C}\) to be an element of the transportation polytope [12], given bywhere \(\mathbf {r} = \mathbf {1} \in \mathbb {R}^N\) and \(\mathbf {c} = \frac{N}{K} \mathbf {1} \in \mathbb {R}^K\) , which corresponds to our equipartition regularization.

Finally, the solution to Equation (4) is equivalent to solving the following problem (up to a constant shift \(- \log N\) ):where \(\mathbf {P} \in \mathbb {R}^{N \times K}\) with entries \(\mathbf {p}_{iy} = - \log (p(y \mid v_i))\) is the cost matrix and \(\langle \cdot , \cdot \rangle\) is the Frobenius dot-product between two matrices.

Note that although we relax \(\mathbf {C}\) to be continuous, the solution to Equation (6) is guaranteed to be integral, which can be obtained in near-linear time using the Sinkhorn-Knopp matrix scaling algorithm [12]. Specifically, we can solve the problem by approximating the Sinkhorn projection of \(e^{\mu \mathbf {P}}\) using the scaling algorithm, where \(\mu\) is a hyper-parameter. In CLEAR, we employ a greedy version of the Sinkhorn algorithm, Greenkhorn [1], to approximate the solution, which is proven to outperform the original version significantly in practice. The cluster reassignment algorithm is given in Algorithm 1. For the details of the Greenkhorn algorithm, we refer the readers of interest to Appendix A.

After obtaining cluster assignments, we further refine the graph topology by strengthening intra-class edges and reducing inter-class connections. Specifically, given cluster assignments \(\mathbf {C}\) , for each edge \((v_i, v_j)\) , we remove it if the probability that \(v_i\) and \(v_j\) fall into the same cluster is less than a threshold \(\tau _r\) , i.e., \(\mathbf {c}_i^\top \mathbf {c}_j \lt \tau _r\) . Additionally, for each node pair \((v_i, v_j)\) , if the probability that \(v_i\) and \(v_j\) belong to the same cluster is greater than another threshold \(\tau _a\) , we add the edge \((v_i, v_j)\) to the graph.

Note that in each iteration, we refine the graph topology based on the original graph instead of the previously refined graph, since informative edges might be accidentally removed at the early stage of the training procedure. Additionally, when adding edges, we consider \((v_i, v_j)\) as candidates only when they connect nodes belonging to the same cluster, i.e., \(\operatorname{argmax}_{k} c_{ik} = \operatorname{argmax}_{l} c_{jl}\) to reduce excessive computation.

The topology refining procedure is designed to increase the purity of the whole graph \(\phi _{\mathcal {G}}\) , which is defined as the probability of an edge in \(\mathcal {G}\) connecting nodes from the same cluster. Formally, we define graph purity asWe can see that topology refining can increase graph purity when the threshold is less than or equal to the current purity, i.e., \(\tau \le \phi _{p}(\mathcal {G})\) . In practice, \(\tau\) is chosen dynamically with \(\tau = \frac{1}{2} \phi _{p}(\mathcal {G})\) .

Our motivation is that learning embeddings for a graph with higher purity will better preserve cluster structures. Considering that embeddings of neighboring nodes are smoothed in graph convolutions, embeddings of nodes belonging to different clusters will become similar due to inter-cluster edges, resulting in indistinguishable node embeddings. We further illustrate this idea through visualization. On the Karate club dataset [49], we conduct topology refining to increase graph purity and add noise edges to decrease graph purity. The learned embeddings are shown in Figure 2. We can see that the modified graph (Figure 2(b)) with higher purity produces embeddings with well-separated clusters, while the clusters are indistinguishable in the graph (Figure 2(c)) with lower purity.

To train the proposed CLEAR model, we first initialize the parameters of GCN by training it with a reconstruction loss [21], which forces the node embeddings to preserve pairwise similarity. Then, we initialize the cluster assignments by employing kMeans on the node embeddings.

After initialization, the node embeddings will be improved in further steps using cluster-aware self-training. Specifically, we iteratively update model weights in three stages, namely, graph representation learning, cluster reassigning, and topology refining. At first, when we perform graph representation learning by solving Equation (3), we fix the cluster assignments \(\mathbf {C}\) . Then, with node representations \(\mathbf {H}\) fixed, we adjust the cluster assignments by solving Equation (6) using the Greenkhorn algorithm. Following Asano et al. [2], to stabilize training, we distribute cluster reassignment and topology refining steps throughout the whole training process. We denote \(\mathcal {S}\) as the set of epochs where cluster assignments will be adjusted. \(\mathcal {S}\) can be chosen freely as long as cluster reassignment is performed at proper intervals. In our implementation, we set updating epoch \(s_i \in \mathcal {S}\) as \(s_i = (E - W) \frac{i}{U + 1} + W, \ i = 1, 2, \ldots , U\) , where U is the total number of reassignment steps throughout training, W is the number of warm-up epochs where cluster reassignment will not be performed, and E is the total number of training epochs. The whole training algorithm is summarized in Algorithm 2.

Datasets. For a comprehensive comparison with state-of-the-art methods, we evaluate our model using four widely used datasets: among them, three are citation networks Cora, Citeseer, and Pubmed, for predicting article subject categories [34, 48], and the other co-authorship network Coauthor-CS (CS) [35] is for predicting the research fields of the authors. In the three citation datasets, graphs are constructed from computer science papers of various subjects. Specifically, nodes correspond to articles and undirected edges correspond to citation links. Each node has a sparse 0/1 bag-of-words feature and a corresponding class label. In the co-authorship dataset, nodes represent authors and they are connected if the two authors have co-authored a paper. Each node has a bag-of-words feature representing keywords for each author’s papers. The statistics of these datasets is summarized in Table 2.

Experimental configurations. We train the model using the Adam optimizer with a learning rate of \(0.01\) . Initially, we train the GCN model with the reconstruction loss for 500 epochs on Cora and PubMed, 250 epochs on CiteSeer, and 200 epochs on CS. Following that, in the self-supervised learning phase, we train the whole model for 15, 60, 50, and 800 epochs on Cora, Citeseer, Pubmed, and CS, respectively. On all the datasets, the optimal transportation solver is run for a fixed number epochs \(E_{ot}\) and with the same hyper-parameter \(\mu\) . Prior to training, we initialize the weight of the encoders by training it with reconstruction loss. Technically, we optimize the loss by negative sampling. It is also possible to initialize the parameters in GCN with a variant of reconstruction loss proposed in VGAE [21]. In our experiments, we initialize our model with the standard reconstruction loss on Cora, Pubmed, and CS, while on Citeseer, we use the variational reconstruction loss.

Hyper-parameter settings. We set the dimension of the node embeddings to 64, the weight decay to 0.0008 in all datasets. For the number of clusters, to avoid trivial solution [6], we set the number of clusters to be around twice the number of ground-truth classes. Specifically, we set the number of clusters to 10, 11, 5, and 20 on Cora, Citeseer, Pubmed, and CS, respectively. For the set of epochs where we perform cluster reassignment, W is set to 10, 80, 20 and, 50 in Cora, Citeseer, Pubmed, and CS, respectively; U is set to 7, 7, 6, and 4 in the four datasets, respectively. Besides, for the optimal transportation solver, \(E_{ot}\) is set to 1,000 and \(\mu\) is set to 20.

To demonstrate the performance of the proposed approach, we first evaluate it on an unsupervised task—we conduct node clustering algorithms on top of the learned node embeddings. In this experiment, we employ kMeans as the clustering method. We run the algorithm for ten (10) times and report the averaged performance as well as standard deviation. We choose two widely used metrics accuracy and Normalized Mutual Information (NMI) in reporting the performance.

Baselines. For a comprehensive comparison, we compare our methods against various unsupervised methods. These methods can be grouped into four categories.

Results and analysis. The performance is summarized in Table 3 with the highest performance highlighted in bold. We report the performance of baselines in accordance with their original papers [14, 52]. From the table, it is evident that our proposed CLEAR surpasses other baseline methods in terms of accuracy on all four datasets. It is worth mentioning that we exceed the existing state-of-the-art model by a large margin of over \(7\%\) in terms of absolute accuracy improvements on Cora.

The results can be analyzed in three aspects. First, we observe that traditional algorithms such as kMeans and spectral clustering, which simply rely on node attributes perform poorly on graph data. Second, conventional network embedding methods outperform traditional clustering methods, but their performance is still inferior to that of attributed graph clustering methods. This demonstrates the power of modern deep learning techniques on graphs that can better leverage both graph structures and node attributes. Last, it is observed that our proposed method outperforms attributed graph clustering baselines by considerable margins. Previous graph clustering methods merely perform node representation learning on the node level, while our method exploits underlying cluster structures in graphs to guide representation learning. Additionally, we utilize cluster information to reduce the impact of noisy inter-class edges, which further improves the quality of embeddings. The improvements show that our proposed CLEAR helps generate embeddings that better preserve cluster structures.

Note that the proposed CLEAR is slightly inferior to AGC on Citeseer in terms of NMI score. However, CLEAR still outperforms it in terms of accuracy and on other datasets by a considerable margin. In all, these results verify the effectiveness of our proposed CLEAR.

We further evaluate the quality of embeddings generated by CLEAR on a supervised task—node classification. After training the model, we conduct node classification on the learned node representations. For a fair comparison, we closely follow the same experimental settings as Gao and Huang [14]. Specifically, we train a linear logistic regression classifier with \(\ell _2\) regularization. For training/test set splitting, we randomly select 10% nodes as the training set and the remaining nodes are left for the test set. Then, we use cross-validation to select the best model. We report the performance on the test set in terms of two widely used metrics, Micro-averaged F1-score (Mi-F1) and Macro-averaged F1-score (Ma-F1). As with the previous experiment, we report the averaged performance and standard deviation of ten (10) runs.

Baselines. In node classification, we include two sets of baseline algorithms: (1) traditional network embedding methods, which only leverage graph structures and ignore node attributes, and (2) attributed graph representation learning methods, which use both graph structures and node attributes. The former category includes representative random-walk-based methods DeepWalk [31], node2vec [17], and GraRep [4]. The latter one includes attributed network embedding methods ANE [19] and DANE [14], and graph neural networks GAE, VGAE [21], DGI [41], AGE [10], and GMI [30].

Results and analysis. We report the performance in Table 4, with the best performance highlighted in boldface. The baseline performance is reported as in their original papers [14]. In general, the results confirm the effectiveness of the proposed method. As shown in the table, the proposed CLEAR performs best on the Cora and Citeseer datasets, compared with state-of-the-art baselines and shows competitive performance on Pubmed and CS, compared with other graph representation learning methods.

As with the conclusions drawn from the experiment of node clustering, traditional network embedding methods such as DeepWalk and node2vec perform worse than deep-learning-based graph representation learning methods, which highlights the importance of incorporating node attributes when training the model. Instead of merely leveraging graph structures, GNNs combine information of graph topology and node attributive information, resulting in better node embeddings. Our proposed method further utilizes cluster information in self-training and refines graph topology by removing inter-class edges that potentially hinder the model from preserving cluster structures in the embedding space. The proposed method produces better node embeddings, so that it achieves significant improvement over existing GNN-based methods.

Note that while DANE is a strong baseline on Pubmed, our proposed CLEAR still outperforms it in terms of accuracy and Macro-F1 score on the other two datasets. We observe that the NMI between cluster labels and ground-truth labels on Pubmed is the lowest among the four datasets, which can help explain the slightly inferior performance of CLEAR on Pubmed. Through the topology refining procedure that is based on cluster labels, our proposed CLEAR may accidentally remove informative edges, which results in performance loss. Recent work DGI and GMI marry the power of contrastive learning into graph representation learning. However, they only optimize node representations in the latent space, which neglect cluster structures that align with the intrinsic properties of the graphs. On the contrary, our proposed CLEAR significantly outperforms them on graph clustering tasks and achieves comparable performance with them in node classification. This can be attributed to our novel approach of leveraging cluster-aware self-training and refining graph topology, which not only enhances cluster structures but also provides more discriminative representations that can benefit node classification tasks. By refining the graph topology and strengthening intra-class edges, our method can generate more informative node embeddings, ultimately leading to improved performance in both tasks.

To further validate the proposed cluster-aware topology refining procedure and justify our architectural design choice, we conduct ablation studies by removing specific components in the topology refining module. Then, we conduct node clustering using the same setting described in previous sections.

Following the ablation study of the proposed topology refining scheme, we further conduct additional experiments using a soft topology refining scheme. For the proposed topology refining scheme, we regard the edge deletion as a “hard” operation, where the intra-class edges will be completely removed for node representation learning. Considering the discrepancy between our model prediction about clusters and ground-truth labels, contrary to hard removal, we may consider an alternative “soft” scheme, where one intra-class edge are reassigned probabilities that express the strength of connection. In this experiment, for each edge \((v_i, v_j)\) , we reassign each intra-class edge with a weight \(\mathbf {A}^{\prime }_{ij} = \mathbf {c}_i^\top \mathbf {c}_j\) ; other edge weights are not modified. Since in the original graph, we represent each edge by \(\mathbf {A}_{ij} = 1\) , our soft modification \(\mathbf {A}^{\prime }_{ij} \lt 1\) , which is able to reduce the connection of intra-class nodes.

The results are shown in Figure 5. We observe that our proposed hard scheme evidently outperforms its soft variant in terms of Micro-F1 and Macro-F1, and performs slightly lower in terms of NMI. The result provides the rationale of using a hard removal scheme. The reason why the soft topology refining scheme performs worse than the hard scheme may be explained from the fact that via the soft removal scheme, there are still many inter-class edges remained, which deteriorate the quality of node embeddings.

In this section, we examine the impact of four key parameters in our model, i.e., the cluster size, the two thresholds for topology refining, and the interval of reassigning clusters. We conduct node clustering on the Cora dataset by varying these four parameters independently. While one hyper-parameter studied in the sensitivity analysis is changed, the other hyper-parameters remain the same as previously described.

Finally, we provide qualitative results by visualizing the learnt embeddings. Specifically, we leverage t-SNE [39] to project the embeddings onto a two-dimensional space and plot them with colors corresponding to the class of each node. The node embeddings are extracted from the penultimate layer of a CLEAR model that is pre-trained on the Cora dataset. For comparison, we also present the visualization of the raw node features in Figure 7(a) as well as embeddings trained with a supervised GCN in Figure 7(b). From the figures, we observe that the representations learned with CLEAR exhibit discernible clusters in the projected two-dimensional space. Note that node colors correspond to seven ground-truth node classes, which shows that the produced embeddings are highly discriminative across seven classes in Cora. Compared to the supervised counterpart, the embedding space learned by CLEAR is more well-clustered, verifying that CLEAR is able to extract and preserve essential information of the graphs.