CLC: A Consensus-based Label Correction Approach in Federated Learning

Label noise generally occurs when human experts are involved. With this claim, possible causes of label noise include imperfect patterns and perceptual errors. Potential sources of label noise can be classified into the following categories [5]: (1) insufficient information: the attributes that are provided to the expert may be insufficient to conclude a reliable label; (2) subjective labeling: there may exist a significant difference in the labeling of experts if the labeling task is subjective such as medical applications or image data analysis; and (3) recording problems: label noise can also be caused by a communication issue or wrong logging such as accidental click. All of these causes make label noise a problem not to be underestimated. The existing body of research on machine learning with label noise suggests three typical categories: (1) loss correction methods, (2) data pre-selection methods, and (3) label correction methods. From a macro point of view, loss correction methods correspond to directly training a robust model, and the latter two methods correspond to detecting and cleaning noise labels. These approaches improve the robustness of the deep neural network to label noise to some extent but remain at different levels of limitation. An example of loss correction methods is the study carried out by, in which an additional softmax layer explicitly models the noise to connect the correct labels to the noisy ones. In a follow-up study, Wang [27] proposes a symmetric cross-entropy with a noise-robust term called Reverse Cross Entropy (RCE). However, these approaches inevitably acquiesce in the participation of all label noise in model training, making it difficult for them to adapt to situations with more label noise. To date, several studies have investigated data pre-selection methods, which refer to selecting clean data before the start of model training; in other words, data with noisy labels is no longer involved in training. INCV [6] finds clean data with multiple iterations of cross-validation, then trains on the clean set. Northcutt [19] explores the use of label transmission matrix to filter noisy labels. However, since the estimate at the initial moment is not always accurate, over-selection or insufficient selection can often happen. Another severe weakness with these approaches is that the dropout of data with noisy labels causes a waste of samples’ available features. To alleviate the loss of information, more recent attention has focused on label correction methods. A significant analysis on label correction was presented by Xiao [28], where directed graphical models characterize the complex noise models. Likewise, undirected graphs are adopted to model label noise. However, much of the researches up to now require support from extra clean data, and the quality of the extra data highly determines whether the inferred noise model matches the actual situation.

Since multiple participants may bring low-quality labeled datasets and the corresponding negative effects on model training, the label noise problem in federated learning has increased academic interest in recent years. Although the main ideas for solving label noise in federated learning are inseparable from the above three mainstream methods, the unique data privacy policy of federated learning makes label noise an open problem. While the raw data is not visible to the server, the noise is no longer easily identifiable. In federated learning, generating intermediate parameters for measuring label noise can relieve this issue. FOCUS [7] reduces the weight of low-quality participants to alleviate the negative effect of label noise. Tuor [26] conducts data-preselection before federated training, which is implemented by estimating the similarity between participants’ training data and the server’s benchmark data. These explorations may be hard to be satisfactory because they drop out the available data features. Also, these methods require a task-related benchmark dataset that should be completely clean. However, most real-world cases can hardly satisfy such requirements; in fact, federated learning is created to solve the problem that high-quality data resources are difficult to upload and share. In our work, noisy labels are corrected via a consensus-based method without benchmark dataset, which means the knowledge hidden in participants’ datasets can be fully utilized.

The simplest efficient consensus algorithm is majority voting (MV) [13], which leads to high-quality labels of instances, as long as the provided label noise is guaranteed to have a certain degree of accuracy. However, real-world scenarios are so complicated that MV does not always work well, especially under situations where biases occur in the labeling process. For the classification task, deciding which class to be adopted completely depends on the number of annotators holding the corresponding opinion, regardless of factors such as the annotator’s expertise. In recent years, several more sophisticated agnostic consensus algorithms such as GLAD, RY have been investigated in crowdsourcing learning [30], highlighting the capability of correctly inferring the minority. GLAD models both the levels of expertise and the difficulties of examples, treating the probability of an object being positive as a latent variable [24]. Raykar proposed the Bayesian approach RY to add specific priors for each class [21]. Unfortunately, these consensus mechanisms are not suitable for the environment of federated learning because the inaccessibility of all data makes them invalid.

Confident learning (CL) [19] is a novel approach that estimates label errors by examining the predictive probability of each given label. By drawing on the concept of self-confidence (given label’s predictive probability), the authors have been able to show that there is a joint distribution between noisy (given) labels and true labels. Unfortunately, CL provides several noise estimation methods which are highly dependent on a benchmark pre-trained model. Besides, CL works by considering all data, which is not intuitively feasible in federated learning. However, if there is a tie among all the participant’s data, and the raw data is not involved in the tie, the true latent label can be identified in a privacy-reserved fashion. We acknowledge that this implementation is a heuristic way of the author’s original algorithm, yet it is justified because dealing with label noise in FL is by no means the same thing as in traditional machine learning settings.

In [25], the authors investigated the learning dynamics of neural networks as they train on single classification tasks. This work defines a forgetting event to have occurred when an individual training example transitions from being classified correctly to incorrectly over the course of learning. Detailed examination of forgetting event showed that the noisy label data is more likely to experience forgetting event. Since counting forgetting event is inconvenient in most cases, a margin tool is designed for estimating the forgetting event. This margin represents the difference between the largest probability among all the classes and the probability of the given class. An interesting regulation concluded by the authors is that frequently forgotten samples usually have a large margin. In other words, this margin tool boosts estimating label noise during the learning course. With that in mind, we take inspiration from this valuable study and deploy a Train-noise splitting strategy (TNS). TNS supports CLC to better adapt to the dynamic federated learning process.

In the context of multi-class data with possibly noisy labels, let \( [m] \) denote \( \lbrace 1,2, \ldots , m\rbrace , \) the set of \( m \) unique class labels, and \( \mathcal {D}^{k}:=(x, y)^{n_{k}} \in (\mathbb {R}^{d},[m])^{n_{k}} \) denote the \( k^{th} \) participant’s dataset of \( n_{k} \) examples \( x \in \mathbb {R}^{d} \) with associated observed noisy labels \( y \in [m] \) . While a number of relevant works address the setting where extra annotator labels are available [4], this paper addresses the general setting where no annotation information is available except the observed noisy labels.

We assume there exists, for every example, a latent, true label \( y^{*} \) . Before observing \( y \) , a class-conditional classification noise process (CNP) [3] maps \( y^{*} \rightarrow {y} \) such that every label in class \( j \in [m] \) may be independently mislabeled as class \( i \in [m] \) with probability \( p(y=i \mid y^{*}=j) \) . In other words, \( p(y=i \mid y^{*}=j) \) describes a label transmission matrix between the given label and true label. CNP is reasonable and currently one of the most popular assumptions for estimating label noise.

In a real-world FL scenario, local datasets unavoidably contain label noise, and the number of noisy labels in participants’ data differs. The global model may overfit noisy data by aggregating model parameters from participants’ sides in federated learning. In the following, we focus on estimating the label transmission matrix over the overall data \( \mathcal {D} = \bigcup _{k} \mathcal {D}^{k} \) . In the traditional distributed learning, since data are distributed in different nodes and are highly controlled by the server, the overall data estimation is feasible. However, in the federated setting, the server can neither access any training data nor obtain an extra task-related benchmark dataset easily. Therefore, we utilize the federated learning’s collaborative nature to offer a Consensus based method for identifying label noise in FL.

According to CNP, our goal is to estimate the random probability of a given label \( y \) could be determined to its true label. This probability is designed for measuring the probability’s lower threshold of assigning the true label to the sample with its raw label. In the literature [19], the term threshold tends to be represented as:

Intuitively, when \( i \ne j \) , the contribution of the sum represents the probability of being determined correctly ( \( p(y^{*}=j \mid y=i) \) ) when the given one is wrong ( \( p(y=i \mid y^{*}=j) \) ); when \( i=j \) , it represents the probability of being determined correctly when the given one is correct. In the traditional machine learning, the overall dataset \( \mathcal {D} = \bigcup _{k} \mathcal {D}^{k} \) can be used to estimate the threshold directly. Assuming that the model \( \boldsymbol {\theta } \) achieves a certain predictive ability (reliable softmax outputs) at \( t \) round, we can obtain:where the term \( \mathcal {D}_{y=i} \) is the set of all samples with a given label \( i \) .

According to CNP assumption, we have:

Using the law of total probability and Bayes, we have:

For traditional distributed learning, Equation (4) is computable as the server can access all the data easily.

Consensus mechanism. Though a threshold \( c_{t}(i) \) can be computed in the traditional distributed learning, it encounters data barriers in federated learning. Thus, we proposed a new consensus method in the FL setting, in which weighted average the local class-wise thresholds to compute the global ones. Following Equation (4), it is necessary here to clarify exactly what is meant by local threshold in each participants’ end, that is, \( c_{t}^{k}(i)=\tfrac{1}{|\mathcal {D}_{y=i}^{k}|} \sum _{x \in \mathcal {D}_{y=i}^{k}} \hat{p}(y=i ; x, {\boldsymbol {\theta }}_{t}^{G}) \) . Similarly, the term that will be used to describe the threshold from an FL server’s view is called global threshold:

For estimating the threshold precisely in FL, we adopt the Weighted Average Consensus method to compute global threshold in a privacy-preserved way. The derivation process is as follows:

The corresponding process of Generating Global Threshold (GGT) is shown in Algorithm 1. Thus far, the global threshold has attempted to provide a reference floor of the probability relating to one label’s predicted probability. And a latent label \( y^{*} \) can be derived as:

By using the class-wise global thresholds, the latent label of a sample can be inferred by two principles, that is, its predicted probability should be larger than (1) the corresponding global thresholds, as well as (2) the other classes. If all the probabilities are unsatisfied with the first principle, the latent label would be set as the second principle only. We illustrate the advantages of the latent label \( y^{*} \) in CLC with actual cases on MNIST dataset [15] in Table 1, and the corresponding digit examples are shown in Figure 2: (1) Effectiveness: for the instance of handwritten digit ‘5’, it is easy to be mislabeled with ‘6’ because of the visual error. In this case, the model is more likely to learn such instances as ‘6’ instead of ‘5’. However, with the Consensus guidance, the model’s average prediction probability for the label ‘6’ is so high that the prediction probability on ‘6’ cannot easily pass. (2) Reliability: for the instance of handwritten digit ‘2’ with its true label, even if the predicted probabilities of multiple classes exceed their global thresholds \( c_{t}^{G} \) , this instance will be easier to identify due to its correct given label. Furthermore, for the instance of handwritten digit ‘8’ with its true label, our algorithm keeps the model’s actual output if no classes surpass the Consensus. With effectiveness and robustness, the latent label \( y^{*} \) will contribute to the next ‘correcting’ stage as an important reference.

Correcting Noisy labels. The Correcting Noisy Labels process (CNL) is presented in Algorithm 2. Before conducting correction, it is important to improve the global model’s predictive ability, which suffers from label noise. To prevent label noise from interfering with model training, we adopt the Train-Noise Splitting (TNS) strategy. Since the global model’s early predictive ability is relatively poor, especially as there is not always an extra benchmark dataset to utilize for pre-training, CLC performs splitting dynamically, that is, splitting before each global round of training. At the participant side, let \( \mathcal {H}_{t}^{k} \) denote the Holdout set for \( t \) round, and let \( {\mathcal {T}}_t^{k}={\mathcal {D}}^{k}\backslash {\mathcal {H}}_t^{k} \) be the Train set. In the early stage of training, since the global model’s performance is still growing up, we adopt a margin tool to alleviate the mistaken holdout. In our setting, whether an instance should be split into \( \mathcal {H}_{t} \) needs to satisfy two conditions as follows:

Only when a sample satisfies \( y \ne y^{*} \) and \( m(x)\gt \tau \) can it be set aside. \( y^{*} \) recalls the temporary latent label shown in Equation (7), and \( m(x) \) can be represented as:

The effectiveness of such margin has been demonstrated in [25], where observed, the data with the noisy label is more likely to have a large margin. By computing the difference between the largest probability among all the classes and the probability of the given class, the noisy sample can be further determined as shown in Figure 3. Let the objective for TNS be that the more clean data is reserved, the better. Figure 4 explains this objective achieved by using margin tool \( m(x) \) . \( \tau \) is a hyperparameter, which is empirically set to 0.1 in the experiment.

Since the holdout datasets \( \mathcal {H} \) for each round may be different, the latent label \( y^{*} \) should be adopted until the global model converges. In our method, the validation loss is used to determine the convergence. In the FL setting, where the local data is inaccessible, the convergence of the model is determined by the average loss on all participant’s validation datasets. For each participant, the validation set is randomly sampled 20% from the training set. Each participant uploads the validation loss to the cloud server. Once the averaged validation loss is no longer getting lower, it is believed to reach convergence. Moreover, a maximum number of iterations is used as an empirical value in other experiments related to the dataset to save time. Such a setting is for the iterative process not to run indefinitely. A similar two-stage training method was also used in [12], and the convergence determination of CLC is consistent with it. Suppose \( T-1 \) round the global model converges, and we introduce the pseudo-label as follows:

Note that if no class exceeds its global threshold, the label will remain as it is.

For a federated learning task, let \( {\mathcal {T}}_t={{\mathcal {D}}}\backslash {\mathcal {H}}_t \) denote the set of samples that was retained before the start of the \( t \) round of training. This set is identified by finding the subset \( \mathcal {T}_{t}^{k} \subseteq \mathcal {D}^{k} \) in each participant. In the first round, the train set \( \mathcal {T}_{t}^{k} \) defaults to \( \mathcal {D}^{k} \) . The overall loss of retained data at participant \( k \) is represented as :where \( |\cdot | \) denotes the cardinality of the set, based on which the global loss across all participants is represented as:

The objective of federated learning on the retained data subset \( \mathcal {T} \) is to find the model parameters \( \hat{\boldsymbol {\theta }} \) that minimizes \( L^{G}(\boldsymbol {\theta }) \) :

The minimization process described in Equation (13) is solved in a distributed manner using a standard federated learning procedure that includes the following steps:

After receiving the global model parameters, each participant \( k \) uses \( \boldsymbol {\theta }_{t}^{G} \) to compute the local thresholds.

We evaluate CLC on four public datasets as listed in Table 2. MNIST [15] is a popular dataset of handwritten digits, which is frequently used in the literature of learning with noisy labels [26]. USC-HAD [31] is a widespread human activity dataset used in the area of ubiquitous computing, which recently attracted the attention of Federated learning-related research [8]. CIFAR-10 dataset consists of 60,000 colour images in 10 classes, with 6,000 images per class. There are 50,000 training images and 10,000 test images. CIFAR-10 [32] is a well-recognized dataset in noise learning, which is often used to verify the generalization of methods robust to label noise [19]. Clothing1M [28] is a large clothing dataset, which comprises 1M images with real noisy labels. In recent years, many settings based on real-world label noise have been proven on Clothing1M [16].

For MNIST and USC-HAD, we partition data into four participants randomly as a simulation of federated learning scenarios. Especially in USC-HAD, a participant’s data is extracted from a specific subject’s sensor data to better simulate privacy issues in federated learning. As these two datasets are clean without noisy labels, we follow the common setting in the literature to add synthetic label noise in the test sets [10]. For CIFAR-10, we randomly divided the training set into 20 parts to verify CLC’s insensitivity to the number of participants. 19 of them were distributed to the participants, and one was used as the benchmark. Note that the participating user settings always meet the requirements of the Cross-silo federated learning (e.g., clients are medical or financial organizations), that is, nofewer than 2 and no more than 100 [14].

Following prior works [19], [23], [9], we verify CLC’s performance on the commonly used asymmetric label noise, where the labels of error-free, clean data are randomly flipped for its resemblance to real-world noise. Specifically, the noisy labels are added according to a randomly generated noise transition matrix \( \boldsymbol {Q}_{y \mid y^{*}} \) presented in Figure 5. We generate noisy data from clean data by randomly switching some labels of training examples to different classes non-uniformly according to \( \boldsymbol {Q}_{y \mid y^{*}} \) . The noise rate is implemented by controlling the trace of the matrix, i.e., the higher the noise rate, the smaller the trace.

The matrix is generated with different traces to run experiments for different noise levels. To make the synthetic process more explicit, we take CIFAR-10 as an example. For the noise rate of \( \gamma \) , we can derive the trace as follows:

Obviously, we can conclude that the following formula is also valid:

Both sides of the equation represent the amount of clean data, and \( N \) denotes the scale of the whole dataset. During the random generation, the element \( q_{i,j} \) of \( \boldsymbol {Q}_{y \mid y^{*}} \) needs to meet the following constraints:

In this way, the generated noise transition matrix \( \boldsymbol {Q}_{y \mid y^{*}} \) can convert the raw label \( y* \) to the noisy one \( y \) :

Here, the element \( q_{i,j} \) of \( \boldsymbol {Q}_{y \mid y^{*}} \) represents the probability of the raw class \( i \) becoming the noisy class \( j \) . The implementation of noise generation has been posted in [19], where the interface can be called directly to generate the required noise matrix.

From the left to right of the heatmap shown in Figure 5, it is apparent that the color on the diagonal line gradually becomes lighter, while the color on the non-diagonal line gradually darkens.

For Clothing1M, there are 37,497 clothing images labeled with both noisy and clean class. Here the ‘noisy class’ refers to the label extracted from the image’s surrounding texts, ‘clean class’ refers to the label manually annotated. We partition these data into four participants randomly, and the amount of real noisy data (’clean class’ not equal to ‘noisy class’) accounts for around 38% of the total data. In addition, we generate a benchmark from the set where all data only has a clean label to provide a benchmark model option.

For experiments on MNIST, we use the network CNN-MNIST as previous works do [2]. By common practice, we use a batch size of 10, a learning rate of 0.01, and update \( \boldsymbol {\theta } \) using SGD with a momentum of 0.5. We use a margin threshold \( \tau \) of 0.1 for the label estimation. In the ablation study, we will show the effect of this hyperparameter.

For experiments on USC-HAD, the input of 300*6 sensor data is not the same situation as the shape of 32*32 images. Inspired by prior works [29], [18], we adopt one-dimensional CNN on this Human Action Recognition (HAR) task. Due to the less correlation between the axes, a 10*1 convolution kernel is used in the CNN architecture to learn the timing features on the sensor data. The successful implementation of CNNs for HAR is due to their capability to utilize convolutions across 1-D temporal sequences so as to capture local dependencies between nearby input samples.

We set the learning rate of 0.001 and used a batch size of 10. SGD’s parameters and the margin threshold are set as the same as MNIST. For experiments on CIFAR10, we adopt the 18-layer Resnet using stochastic gradient descent with the momentum of 0.9, weight decay of 0.0005, and batch size of 64. Here, the hyperparameter \( \tau \) is also set to 0.1. For all experiments, the number of local epochs is 20, and the number of global epochs is 200.

For experiments on Clothing1M, we follow previous work [20] and use the ResNet-50 pre-trained on ImageNet. We use a batch size of 64, a learning rate of 8 \( e^{-4} \) , and update \( \boldsymbol {\theta } \) using SGD with a momentum of 0.9 and a weight decay of 9 \( e^{-3} \) . The hyperparameter is set as 0.1.

To our best knowledge, there are few works that correct noisy labels in federated learning, and few works dealing with noisy labels not requiring a task-related benchmark dataset. We compare CLC performance versus state-of-the-art approaches, respectively, under the premise of with benchmark model (BM) and without a benchmark model (No BM). Note that all these baselines, including traditional machine learning (ML) approaches, have been transferred to the context of federated learning:

From Table 3, it can be seen that by far, few works claim that they can address label noise without a benchmark model. Besides, the outstanding FL approach DS was implemented by exchanging the sample-wise intermediate parameters, exposing a lot of potentially sensitive information. Our method CLC only exchanges class-wise information to preserve data privacy. Moreover, CLC deploys a Consensus-based label correction approach in FL to utilize the cooperative nature of FL and alleviate knowledge loss. In addition, CLC does not require complex hyperparameters; generally, a margin threshold of 0.1 can deal with lots of cases.

We report the performance of CLC from the following two aspects:

Versions of CLC. We conducted several ablation experiments with noise rates of 0.3, 0.4, and 0.5 on CIFAR-10 to make the consensus mechanism more persuasive. The ablation study does not use the benchmark model, and the number of participants is set to 19. We analyzed CLC with different crucial parts, i.e., the iterative holdout (H), the label correction (C), and the class-wise thresholds aggregation part (agg). The experimental results are shown in Figure 10. Without the agg part, the labels were corrected based on the local thresholds. Since CLC deals with label noise, it outperformed FedAvg under all the settings. As the noise rate increases, the advantages of ‘CLC: H+C+agg’ become significant. The aggregated global thresholds, known as the product of the consensus, can better mine the noise distribution from a global perspective. Moreover, the label correction can further augment the available training data. Overall, the H part helps filter the noisy labels, and the C part reuses them, while the agg part helps learn a better distribution of noise and avoid negative correction. Hyperparameters. We conduct an ablation study to examine the effect of the hyperparameter \( \tau \) . \( \tau \) is the lower threshold of margin \( m(x) \) used to determine further the size of the holdout dataset \( \mathcal {H} \) . We experiment with \( \tau =0, 0.1, 0.3, 0.5 \) . Figure 11 shows the performance of CLC using different \( \tau \) trained on USC-HAD with the noise rate of 0.4. Closer inspection of Figure 11 shows that the optimal \( \tau \) is generally fixed at 0.1. The corresponding history recording of margin distributions are presented in Figures 12, 13, 14, and 15, respectively. The margin distribution here reflects the margin of those data that are considered to contain label noise. Note that \( \tau =0 \) means that noise detection is conducted without the margin tool. As shown in Figure 12, during the period before noisy labels are corrected, although the detected noise (true positive) is gradually increasing, a lot of clean data (false positive) is mixed. Interestingly, a large amount of noisy data and a certain amount of clean data gradually move closer to larger values. Our goal is to find as much noisy data as possible and minimize clean data mixing. We study \( \tau \) from 0.1 to 0.5, taking a step of 0.2. Figure 13 below illustrates that when \( \tau =0.1 \) , the number of false-positive samples decline sharply, and the number of true positive stays relatively stable–noted that CLC aims to revise noisy labels as much and as accurately as possible. Unfortunately, with \( \tau =0.3 \) , the number of true positives and false positives both drastically reduce. Furthermore, the margin threshold of 0.5 in Figure 15 performs even worse than 0.3. In summary, these results show that a margin threshold of 0.1 guarantees both recall and precision.