Meta-learning Advisor Networks for Long-tail and Noisy Labels in Social Image Classification

We developed a new advisor network that helps the deep neural network (DNN) to address both the noisy labels and long-tail image classification problems. Our method is composed of two main parts that can work jointly: Meta Feature Re-Weighting (MFRW) and Meta Equalization Softmax (MES). The first component, MFRW, makes use of an auxiliary advisor network that automatically learns how to weigh the features extracted from a DNN during its training. Our idea was to exploit an attention mechanism to enhance the useful parts of visual information and lower the rest. If a network can concentrate only on some convenient parts of an image, that information can contribute to increasing the overall generalization capacity of the network even if the annotation is wrong. This is also true for the long-tailed distribution of data where information from common classes can be leveraged to improve performance on the rare ones. In the second component, MES, the advisor network automatically learns how to reduce the discouraging gradients of some images concerning others. In long-tailed distributions, the discouraging gradients of frequent categories samples can worsen the learning of the rare ones [46]. This can happen even with noisy labels because the discouraging gradients of an example with the wrong label can affect the correct learning of the entire classifier. These two methods can be used simultaneously to help the learning of an image classifier, reducing the negative effect that unbalanced or noisy annotations in the training datasets can produce. Our advisor network is trained with the meta-learning paradigm, so it can know the current state of the classifier and learn how to help it at that moment.

We first introduce a meta-learning basic formulation for methods that learn robust deep neural networks from noisy and long-tailed category distributions. We then proceed to show in detail each part of our method: Meta Feature Re-Weighting is specified in Section 3.3 and Meta Equalization Softmax in Section 3.4. Finally, the learning process of the classifier together with the advisor network is described in Section 3.6.

In general, meta-learning (ML) refers to the process of improving a learning algorithm over multiple learning episodes, it is also called learning to learn. ML is divided into two algorithms: an inner (or base) and an outer (or upper/meta) algorithm. The inner one solves the main task minimizing an objective function, for image classification we have a convolutional neural network and the cross-entropy loss, respectively. Instead, the outer algorithm updates the inner one such that it improves also on an outer objective function. When the objective functions are the same for both algorithms, the outer algorithm can help the inner one to work well on new data distribution. If the new distribution is a smaller version of training data but free of errors and balanced, it is possible to train the outer algorithm to solve the problem of noisy or imbalanced labels inside the main training data. We refer to this distribution as meta-set. As in [45], the outer algorithm can be a multilayer perceptron network, called meta-model, that learns automatically how to address these problems helping the main image classifier during its learning. We introduce the symbols useful for understanding ML in this particular setting and how the entire learning process is divided, describing the [45] algorithm for simplicity.

Let \(D^{train} = \lbrace x_i^{tra}, y_i^{tra}\rbrace ^N_{i=1}\) be the training set with noisy or imbalanced annotations, where \(N\) is the total number of samples, composed of an image \(x_i\) and the correspondent one-hot label \(y_i\) over \(C\) classes. The main DNN model is defined as \(\Phi (\cdot ; w)\) , where \(w\) are its parameters. The prediction on an input image \(x\) is \(\hat{y} = \Phi (x; w)\) and the optimal parameters \(w^*\) are obtained by minimizing the softmax cross-entropy loss \(\ell (\hat{y}, y)\) on the training set. Let \(D^{meta} = \lbrace x_j^{meta}, y_j^{meta}\rbrace ^M_{j=1}\) be the meta-set, a well-verified and balanced version of training one but much smaller, \(M \ll N\) . The meta-model is defined with \(\Psi (\cdot ;\theta)\) , parameterized by \(\theta\) . In [45] the optimal parameter \(w^*\) is derived using a loss weighted with a value predicted by the meta-model. The meta-model is trained minimizing the softmax cross-entropy loss of previously updated \(\Phi (\cdot ;w^*(\theta))\) on the meta dataset \(D^{meta}\) .

Both \(\Phi (\cdot ; w)\) and \(\Psi (\cdot ;\theta)\) can be updated by alternating optimization through gradient descent. An online strategy, that is divided into three main steps, can be adopted to update \(\theta\) and \(w\) through a single optimization loop. This guarantees the efficiency of the algorithm and its convergence [45]. In the first step, called Virtual-Train, the original DNN will not be updated and the optimization is carried out on a virtual model that is the clone of the original one. Keeping in mind the virtual update previously carried out, in the successive step called Meta-Train, the meta-model is updated. Actual-Train is the last step where the base DNN model is optimized taking into account the already updated meta-model.

Human attention is the ability of the brain to selectively concentrate on one aspect of the environment while ignoring other information. Attention for a DNN is a mechanism that tries to mimic the cognitive attention of the human brain, calculating a soft (or hard) mask which is then multiplied with the visual features of the network. The mask \(W\) is usually the output of a function \(g\) of some input \(x\) and \(W\) is element-wise multiplied with a feature \(f\) of the networkwhere \(\odot\) is the symbol for element-wise multiplication.

This intensifies the important parts of the feature and reduces the rest. We proposed a meta-attention mechanism, called Meta Feature Re-Weighting (MFRW), that can be used to mitigate noisy or imbalanced labels problems in the training data. In the first case, if there is a mismatch between the content of the image and its associated annotation, that can lead to a degradation of the classifier’s performance for that annotated class. With our method, the main network can use only parts of the erroneous visual information to improve performance in that class. Instead in the case of imbalance, MFRW can attribute to the visual information relative to common classes smaller importance than those of the rare ones. Finding the handwritten function \(g\) that generates the right masks for each of the two cases is challenging. We used a meta-model to automatically infer the correct \(g\) . This gives two properties to \(g\) : it can change during the training of the main network and it can adapt automatically to the problem present in the training data. The element-wise product is done between the feature \(f\) extracted from a DNN and a vector of weights \(W_f\) learned from a meta-modelThe meta-model can take into consideration important aspects of each training data, so it can generate the proper activation weights based on them. Attention must be differentiated between the various categories, as each may have a different number of examples and noise levels. This is done by giving as input to the meta-model visual features extracted from the classifier’s backbone. In the case of unbalanced labels, examples of the more common categories, since they are presented many more times during training, are easier to be learned than those of the rarer ones. In addition, mislabeled images have a different difficulty than cleanly labeled images, as they are outside the correct distribution of each category, and their size is often smaller than the correct ones. The attention should be adjusted according to the difficulty that a training example represents for the classifier. This way, it can focus differently on the information in the data by learning a better representation. The cost value typically used to express the difficulty of classification samples [26] is given to the meta-model in combination with the visual features.

The conventional loss function for image classification is the softmax cross-entropy. A multinomial distribution \(p\) over \(C\) categories is obtained from the network outputs score \(z\) with the softmax activation function. Then the cross-entropy is calculated between \(p\) and target distribution \(y\) . The softmax cross-entropy loss can be formulated as:where the distribution \(p_j\) is described as follows:

When the distribution of categories in the training dataset is imbalanced, the softmax cross-entropy loss makes the learning of rare categories easily suppressed by the common ones. In [46] a softmax equalization loss is proposed to avoid discouraging gradients from samples of frequent categories for the rare ones. The difference with the softmax cross-entropy is the weighting of a term within the softmax activation function. The new distribution \(p_j\) is calculated as:where

The element \(T_\lambda (f_k)\) is a handcrafted threshold function that outputs a value \(\in \lbrace 0,1\rbrace\) based on the category frequency value \(f_k\) . When \(T_\lambda\) output is 1 the gradient is ignored, otherwise it is taken into account. Instead, \(\beta\) is a Bernoulli random variable with a probability of \(\rho\) to be 1 and \(1 - \rho\) to be 0.

The strategy of avoiding the discouraging gradients can be useful in other problems different from imbalance training, for example image classification with noisy labels. The discouraging gradients of a mislabeled image can be scaled to not harm the correct learning of the classifier model. This behavior can be obtained by modifying the weights \(\tilde{w}_k\) passed to the \(EQ_{Softmax}\) function in the Equation (6). The element that determines each category’s weight is \(T_\lambda (f_k)\) , but it works only for the imbalance training problem. Writing a new function for the noisy annotation problem is hard because the noise can be really complex or completely unknown, for example when data is collected automatically [52].

Inspired by this we proposed a meta-learned equalization loss (MES) that can adapt the weight to the task that needs to be solved. The new formulation of the weights \(\tilde{w}_k\) passed to the \(EQ_{Softmax}\) is:where \(s_k\) is the vector of value \(\in (0,1)\) . This vector \(s_k\) is the output of a meta-model trained to help the main model handle noise and imbalance label problems present in the training data. The visual feature and the cost of each training data are given as input to the meta-model. This allows the model to generate output vectors \(s_k\) that are differentiated between classes and between “easy” and “hard” examples.

Since both MFRW and MES require the same input data, it was possible to combine the two methods through a single meta-model. Our meta-model \(\Psi\) is a neural network composed only of a fully connected layer. The inputs are a feature \(f\) and a loss value \(\mathcal {L}_x\) . Each input is projected in a fixed-size embedding space through a separate fully connected layer followed by a ReLu function. These embeddings are concatenated to form a larger common space, the size of which is the sum of the dimension of each previous embedding. MFRW method requires a weight vector \(W_f\) in the range \(\in (0,1)\) of the same size as the feature input \(f\) . Instead, MES needs a weight vector in the range \(\in (0,1)\) but with a length equal to the number of classes \(C\) . From the last embedding space, we get the needed outputs thanks to a fully connected layer followed by a sigmoid activation function for each of the outputs. In this way, MFRW and MES can learn a common internal representation from the inputs, obtaining their respective benefits.

In this section, we describe how the base classifier \(\Phi\) and our meta-model \(\Psi\) are trained jointly. Because the meta-model needs as input the visual feature, we separate the main model \(\Phi (\cdot ; w)\) into two different parts: the backbone \(\Phi _b(\cdot ; w_b)\) and the category predictor \(\Phi _c(\cdot ; w_c)\) . The first one has an image \(x\) as input and gives out a feature vector \(f\) . Instead, the second part has \(f\) as input and a probability score vector \(z\) as output. In this way, it is possible to manipulate the feature \(f\) directly with our meta-model \(\Psi\) . The meta-model takes two different inputs \(\Psi (f,\mathcal {L})\) and gives back two vectors of weights \(W_f\) and \(s_k\) . Our algorithm is divided into four main phases that are shown in Figure 2 and summarized in Algorithm 1. We describe our method in detail starting with the \(t\) -th iteration and moving forward each step until we reach the \((t+1)\) -th. Different from the meta-learning optimization strategy described in Section 3.2, we need an additional initial phase, called Loss Pre-Calculation (Figure 2(a)). The value of loss \(\mathcal {L}^{pre}\) related to the training batch \(X^{train}\) must be calculated at the beginning. This loss value must be dependent on the original feature \(f^{train}\) and not on the weighted one \(f^{att}\) . In the second step Virtual-Train (Figure 2(b)), \(\Phi _b^t\) and \(\Phi _c^t\) are the virtual clones of the backbone \(\Phi _b(\cdot ; w_b)\) and the category predictor \(\Phi _c(\cdot ; w_c)\) at the beginning of the \(t\) -th iteration. We obtain the features \(f^{train}\) passing through \(\Phi _b^t\) the batch \(X^{train}\) . Then the loss values \(\mathcal {L}^{pre}\) pre-calculated and its relative feature \(f^{train}\) are given to \(\Psi ^t\) (the meta-model at time \(t\) ) to obtain the two vectors of weights \(W_f\) and \(s_k\) . The feature \(f^{train}\) is multiplied element-wise with \(W_f\) to get a new feature vector with attention \(f^{att}\) as in Section 3.3. The modified feature is passed to the predictor \(\Phi _c^t\) obtaining the score \(z^{train}\) . Now we calculate the \(\mathcal {L}^{train}\) with the equalization loss, described in Section 3.4, using the vector \(s_k\) in the Equation (8). Then \(\Phi _b^t\) and \(\Phi _c^t\) parameters are virtually updated to minimize \(\mathcal {L}^{train}\) , excluding those of \(\Psi ^t\) . For the third step Meta-Train (Figure 2(c)), we need a clean and balanced meta-dataset that will be used to train the meta-model \(\Psi\) . We pass a meta batch \(X^{meta}\) through the virtually updated \(\Phi _b^{t+1}\) and \(\Phi _c^{t+1}\) in order to get a validation loss \(\mathcal {L}^{meta}\) . In this step, the feature is not modified and the loss is the classic softmax cross-entropy loss. Then only \(\Psi ^t\) is updated minimizing \(\mathcal {L}^{meta}\) . In this way, the meta-model is optimized to help the main model minimize its error on clean and balanced data. Here the optimization takes into consideration also the previous Virtual-Train. In the last phase, Actual-Train (Figure 2(d)) the original \(\Phi _b^t\) and \(\Phi _c^t\) are optimized taking into account the updated meta-model \(\Psi ^{t+1}\) . Our meta-model is used only during the training of the main network \(\Phi\) when external help is needed to solve noisy or imbalance label problems. It is discarded at test time when only the main network is retained as the final model.

Excluding the Loss Pre-Calculation phase, the Virtual-Train, Meta-Train, and Actual-Train steps need a backward operation in addition to a forward one. The Meta-Train backward step, in which the meta-gradient is computed from the loss on the meta-set, takes more than \(80\%\) of the total computation [53]. In this step, to update the meta-model parameters, the meta-gradient is back-propagated backward through each layer of the main network. Since normal training does not involve this step, this additional cost quickly becomes significant as the number of layers in deep networks increases. In addition, the amount of GPU memory required is duplicated compared to traditional training. The gradients obtained in the Virtual-Train step must still be kept in memory so that the meta-gradient can be calculated during the Meta-Train step. These computation and memory problems are typical of a lot of meta-learning approaches. However, a method like [53] which computes the meta-gradient with a faster layer-wise approximation provides strategies to overcome them. These overhead costs are present only during the training. In the test phase, the meta-model is not used and there is only a forward pass on the classifier to get a prediction on the input.

Following previous works [20, 44, 45], we used CIFAR-10 and CIFAR-100 as bases to generate synthetic datasets. They are composed of \(50,\!000\) training images and \(10,\!000\) test images of size 32 \(\times\) 32. Off the training set, we randomly selected 100 images for CIFAR-10, and 10 images for CIFAR-100, per class to create the meta-set for meta-training. The long-tailed versions of the datasets, CIFAR-LT-10 and CIFAR-LT-100, are created randomly removing training examples [3]. Following the standard evaluation protocol for the long-tailed problem, we studied five different imbalance factors (IFs) of 200, 100, 50, 20, and 10, where IF = 1 coincides with the original datasets. These IFs are related to the parameter \(\mu \in (0, 1)\) , where the number of examples dropped from the y-th class is \(n_y\mu ^y\) and \(n_y\) is the original number of training examples for that class.

We tested our method also on a noisy labels version of CIFAR-10 and CIFAR-100, namely Flip CIFAR-10 and Flip CIFAR-100. In these variants, we chose the standard Flip (or asymmetric) noise because it is designed to mimic the structure where labels are only replaced by similar classes, e.g., dog \(\leftrightarrow\) cat. This type of noise usually happens when there is ambiguity between categories or visual similarity between images [52]. The noise ratio is controlled with a parameter \(p\) , which represents the probability that a correct label is flipped with the corresponding similar one. In this way, we could test our method on different levels of noise, from \(p = 0.0\) (no noise) to \(p=0.6\) (heavy noise).

Merging the two strategies to inject data issues, we also introduced a new synthetic version of each dataset, named respectively LT Flip CIFAR-10 and LT Flip CIFAR-100, as a new evaluation protocol for the case of training data with simultaneously unbalanced and noisy labels.

In [33] a long-tailed version of ImageNet-2012 [10] called ImageNet-LT, was introduced as standard evaluation protocol for the long-tailed problem. From a Pareto distribution with shape value \(\alpha = 6\) , each class size is sampled to obtain the corresponding number of images for each one. ImageNet-LT has \(115,\!800\) training images in \(1,\!000\) classes with an imbalance factor of \(1280/5\) . We randomly selected 10 images per class from the provided validation set to create our meta-set for meta-training. The test set is the original balanced ImageNet-2012 validation set with 50 images per class.

The Places-LT dataset is created by sampling from the dataset Places-2 [58] with the same strategy used for ImageNet-LT. The training set is composed of \(62{,}500\) images from 365 classes with an imbalance factor of \(4980/5\) . The test set has 100 images per class. Our meta-set is created by randomly selecting 10 example per class from a validation set of 20 images per class.

The Clothing1M [52] is a dataset that is composed of 1 million images of clothing taken from online shopping websites. There are 14 categories like T-shirts, Shirts, Knitwear, and so on. The labels are obtained from the text of the images provided by the sellers and not from an expert annotator. This process introduces into the labels a real-world noise, which cannot be predicted in advance. A validation set of 14,313 manually well-annotated images is provided and it was used as the meta dataset in our experiments.

In every experiment, the meta-model was optimized with Adam [24] and a learning rate of 1e-4. The size of each embedding space was set always to 100. The probability of \(\rho\) of the Bernoulli distribution \(\beta\) for MES was equal to 0.9.

We conducted several experiments on the imbalance training problem related to image classification. We tried different settings and datasets to compare our method with the others in the literature. We tested MFRW and MES both disjointly and together (MFRW-MES). We consider as baseline method the direct training of the classifier with a standard softmax cross-entropy loss.

The first part of experiments on CIFAR-LT-10 and CIFAR-LT-100 was conducted with a Resnet-32 network trained through SGD with a momentum of 0.9, weight decay of 5e-4, batch size of 128, and a starting learning rate of 0.1. The learning rate decreased to its \(1/10\) at the 160 epoch and 180 epoch, stopping the learning at 200 epochs. The results of our methods and related works are shown in Table 1.

For the CIFAR-10-LT our methods MFRW and MFRW-MES got the first and the second-best accuracy values, especially at higher values of IF. Instead, in CIFAR-100-LT the higher accuracy results are shared across BALMS [43] and our method MFRW-MES.

Increasing the number of categories to be classified from CIFAR-10-LT to CIFAR-100-LT, but maintaining the same ResNet-32 backbone, the gain in performance of MFRW was less pronounced than MES. Instead, the few classes of CIFAR-10 led MES to a modest improvement when compared to MFRW. These behaviors might depend on the number of examples per class or even on the classifier backbone.

To investigate further on that we conducted a second phase of experiments where a more strong preprocessing (increase the variety of training samples) and a different learning rate scheduler were applied to the training. A Resnet-32 classifier is trained for \(13,\!000\) iteration with a batch of 512, on which was applied AutoAugment [8] and Cutout [11]. The initial learning rate was 0.1, then decreased to zero with a Cosine Annealing scheduler [34]. The optimizer used was SGD with a momentum of 0.9 and a weight decay of 5e-4.

The results of this experiment are reported in Table 2. It shows an overall performance improvement over the results in Table 1 due to the additional preprocessing applied to the inputs of the classifier. In this setting our method MFRW-MES, which benefits jointly from MFRW and MES strategies, got comparable results with BALMS [43]. Compared to the results obtained in Table 1, it can be seen that MES benefits more from the increase than MFRW in both the CIFAR-10-LT and CIFAR-100-LT datasets, mainly in the highest imbalance factor IF = 200. This suggests that the few classes of CIFAR-10 and the very small number of training samples, as in Table 1 (where there is no data augmentation), led MES to a confused estimation of the vector \(s_k\) used in Equation (8). Because MFRW modifies the visual feature size of the classifier, the small number of parameters of the Resnet-32 architecture could be a limitation for this method. For this reason, we also tested our methods on the backbone ResNet-18 with more parameters (11.17M trainable parameters) and a bigger visual feature size than ResNet-32 (0.48M trainable parameters), but with the same settings as the experiments done in Table 1. The new visual feature size went from 64 of ResNet-32 to 512 of ResNet-18.

The results in Table 3 demonstrate our intuition that MFRW benefits from a bigger classifier backbone. Instead, MES achieved only a small improvement in having a more complex backbone showing that its performance is related to the number of examples in the training set and their preprocessing. With Resnet-18 as backbone, our method MFRW-MES and MFRW got the first and the second-best accuracy result on almost every IF value. Figure 3 shows how the accuracy of MFRW-MES and BALMS [43] varies with the number of parameters of the classifier’s backbone on CIFAR-10-LT (3(a)) and CIFAR-100-LT (3(b)) with IF = 100.

From Tables 1–3 it is possible to notice how our method exceeds or is in line with the results obtained from the state-of-the-art algorithms for long-tailed training. We obtained the best accuracy values in almost all IFs, especially when the dataset is extremely unbalanced (IF = 200,100,50). Table 3 shows the effectiveness of our method with a bigger network backbone ResNet-18. Both MFRW and MES obtained good results, even individually. We could observe how they could be used simultaneously without compromising the final performance of the classifier. We designed MFRW-MES to address even the more complex case of imbalance together with noisy labels.

The embedding space utilized in the meta-model employed by MFRW-MES is learned by taking into account the collaboration between MFRW and MES. During the training of MFRW-MES, the embedding space is influenced first by MES, which acts directly on the loss of the classifier, and then by MFRW, which operates on the visual feature. In Table 1 for the case of CIFAR-10-LT, where MES has poor performance, the application of confused predicted vectors \(s_k\) to the loss let MFRW received a diminished gradient on the visual features, rendering it incapable of learning the \(W_f\) masks correctly. In some cases, this makes MFRW-MES having a lower accuracy result than the application of the singular method MFRW. Instead, in Table 3, where a more capable backbone Resnet-18 (11.17M trainable parameters) with a bigger visual feature size than ResNet-32 (0.48M trainable parameters) was used, MFRW-MES got better accuracy results than each singular method MFRW and MES. In this case, when the two methods were applied jointly, MFRW could act upon the visual features even if it received a reduced gradient (produced by the application of MES), exploiting the bigger number of learnable parameters of the backbone.

Following the experiment setup of [43], we employed ResNet-10 and ResNet-152 networks for ImageNet-LT and Places-LT, respectively. For ImageNet-LT, we adopted an initial learning rate of 0.2 and decayed with Cosine Annealing scheduler during training of 180 epochs. For Places-LT, the learning rate started at 0.005 and it was reduced like for ImageNet-LT. We trained ResNet-152 for a total of 60 epochs with a batch size of 64. In both cases, our method started from a baseline that had been pre-trained on the entire dataset. We did not freeze the feature extractor part of the pre-trained network as the decoupled training strategy of [21] does. We pre-trained the backbone to accelerate the total training time and to make our method starts from an almost good feature extractor.

Table 4 shows the result of MFRW-MES on ImageNet-LT and Places-LT. In the first dataset, our method achieved a Top-1 accuracy value comparable to other methods that only target this task. Instead, for the Places-LT dataset, we got the second-best result.

With these experiments, we showed how our algorithm can solve the long-tail data problem via a simple advisor network trained with the meta-learning paradigm.

We trained our model under Flip (or asymmetric) label corruption noise at various levels. To assess the performance of the advisor network, we compared it to other works that studied this type of noise. We trained a ResNet-32 through SGD with a starting learning rate of 0.1 and batch size of 128. We decreased the learning rate at epoch 50 and 70 by a factor of 0.1. We stopped the training after 100 epochs. We also reproduced the [43] algorithm under this experiment setting to observe how an ad-hoc long-tailed distribution method works under the Flip noise. The baseline method is a direct training of the classifier with a standard softmax cross-entropy loss.

We can notice from Table 5 that our method obtained the best results for the flip noise on CIFAR10 and CIFAR100. The use of our advisor network avoided a drastic accuracy drop than the other methods, especially when the noise was really strong ( \(p = 0.6\) ). When there is no noise ( \(p = 0.0\) ) our method got worse accuracy values than a normal training with the classic softmax cross-entropy loss on both CIFAR10 and CIFAR100. It happens because the advisor network, trying to help the classifier, introduces a bias of the examples distribution contained in the meta-set. If the training distribution already reflects the test one better than the one contained in the meta-set then, introducing this meta bias, the accuracy is a little worse than without. In some experiments, MES may slightly outperform MFRW-MES. This behavior is reasonable because MFRW-MES, which tries to address even the more complex case of imbalance and noisy labels, attempts to give the classifier model well-balanced training.

We decided to introduce a new synthetic dataset setting in which unbalanced and noisy label problems are both present. We chose three values of IFs (200, 100, 10) and two of \(p\) (0.4, 0.6), and all possible combinations for both CIFAR10 and CIFAR100 were generated. All experiments were performed by training a ResNet-32 with the same settings and hyperparameters as the one used to obtain the results listed in Table 1. This experiment is important to establish the ability of an algorithm to handle different types of dataset conditions at the same time.

We compared our method with BALMS [43] because it is designed for long-tailed distributions, and with MW-Net [45] and GDW [5], which can deal with any type of problem in the data, similar to us. The results shown in Table 6 indicate that our advisor network can manage at the same time both noisy labels and long-tailed distributions better than the other methods. MFRW-MES can exploit the two different properties of MFRW and MES at the same time, achieving better results than each method used separately.

In order to test real-world noise, we used Clothing1M and ResNet-50 as backbone, pre-trained on ImageNet, which was trained through SGD with a momentum of 0.9, weight decay of 1e-3, and a starting learning rate of 0.01. The batch had a size of 32 and it was preprocessed by resizing the image to 256 \(\times\) 256, then random cropping a 224 \(\times\) 224 patch, and finally performing normalization. The total training process consisted of 20 epochs where the learning rate was multiplied by 0.1 after 10 and 15 epochs.

The results reported in Table 7 show how our method obtains the state-of-the-art accuracy on the clothing dataset, improving it by \(3,10\%\) compared to the best algorithm previously used [39]. We got a \(12.33\%\) increment to the baseline accuracy.

To verify the effectiveness of our method in a mixed setting of real-word label noise and long-tail distribution, we apply three different levels of IF (1, 50, 100) to Clothing1M as described in [22]. We trained a ResNet-18, pre-trained on ImageNet, with the SGD optimizer with a momentum of 0.9, weight decay of 1e-4, and a starting learning rate of 0.01. We optimized the classifier for a total of 20 epochs, and the learning rate was multiplied by 0.1 after 10 and 15 epochs. We used a batch size of 64 that was preprocessed by resizing the image to 256 \(\times\) 256, then random cropping a 224 \(\times\) 224 patch, and finally performing normalization.

From the results in Table 8, we can observe how our method obtains state-of-the-art accuracy values on the Clothing1M dataset with and without an artificially applied long-tail distribution.

We investigated how our softmax weight \(s_k\) , learned with MES, differs from the handmade solution proposed by [46]. We measured the effectiveness of each solution by calculating the Mean Absolute Error (MAE) between the distribution of the class sizes, normalized between 0 and 1, and the vector of the weights passed to Equation 6. Figure 9 shows the MAE values obtained during the training of the main network on CIFAR-LT-100 with different IF values. MES fits the target distribution better than the simple threshold function applied in [46] for all IF values and does not need any extra hyperparameter tuning.