2.1 Deep Active Learning
DAL methods are divided into query-synthesizing or query-acquiring approaches, as summarized in the survey [
19]. The query-synthesizing approach uses generative models to synthesize informative samples. The query-acquiring approach recommends the most informative images from the unlabeled image set. This approach has been intensively investigated; therefore, we focus on the query-acquiring approach in this article. Table
1 presents a summary of the query-acquiring DAL methods. The best-performing method in each category is shown in bold.
Uncertainty approach: This approach measures uncertainty based on classification probabilities and recommends the most uncertain images from the unlabeled dataset [
24–
28]. These images are expected to capture the most informative features that are not already present in the labeled dataset. Adding the newly labeled images verified by the domain expert improves the classification performance.
Cost-Effective Active Learning (CEAL) [
26] follows the uncertainty approach and recommends unlabeled images based on a classification probability threshold. Furthermore, CEAL does pseudo-labeling on unlabeled samples with a very high-class prediction probability over another threshold that is automatically adjusted for each iteration. These images are included in the training dataset for the predicted class for the next iteration without domain expert verification. Pseudo-labeling was shown to improve CNN classification performance.
Active Incremental Fine-Tuning (AIFT) [
28] utilizes data augmentation for recommending unlabeled images. AIFT is based on an observation that all images synthesized from the same image share the same label and are naturally expected to have similar predictions by the image classifier. AIFT recommends unlabeled images based on the authors’ definitions of entropy and diversity measures on some synthesized images of each unlabeled image. AIFT does not perform well under class imbalance [
14]. When the images of the common class are much more diverse than those of the rare class, the entropy and the diversity measures are biased toward recommending images of the common class.
Pretext Tasks for Active Learning (PT4AL) [
29] utilizes self-supervised pretext tasks and an uncertainty data sampler to recommend images for human labeling. PT4AL does not require the initial manually curated labeled dataset and is applicable for several main tasks, e.g., image classification and semantic segmentation. The method trains a pretext task model on the unlabeled images with a rotation prediction loss that is highly correlated to the main task loss, i.e., image classification loss. Then, PT4AL sorts the unlabeled images in descending order of the loss values and recommends the images with the highest losses for the initial seed dataset. For recommendation in a subsequent iteration, PT4AL uses an uncertainty measure based on the posterior class probability of the remaining unlabeled images predicted by the image classifier. PT4AL uses a batch splitter to ensure balanced sampling across the entire data distribution. PT4AL can handle a mildly imbalanced dataset well [
29]. However, our experimental results show that PT4AL does not perform as well for moderately to severely imbalanced classes.
The authors of [
30] proposed a task-agnostic active learning method. They attached a small parametric module called the “loss prediction module” to the image classifier to learn to predict the losses for the unlabeled images. Then, this module predicts the unlabeled images that are likely to fail correct classification by the image classifier. The loss prediction module is simple and applicable to any neural network regardless of the task, the number of tasks, or the complexity of the main task architecture. However, like CEAL and AIFT, the loss prediction module is not designed for class imbalance.
Deep Bayesian Active Learning (DBAL) [
31] utilizes Bayesian convolutional neural networks [
32] to build an active learning framework using graph-based approaches to handle high-dimensional image data. DBAL first performs Gaussian prior modeling on the weights of a CNN and then uses variational inference to obtain the posterior distribution of the network prediction. DBAL also uses the Monte-Carlo dropout stochastic regularization technique [
33] to obtain posterior samples. DBAL achieved good accuracy on real-world datasets. However, it is unsuitable for large datasets due to the need for batch sampling [
27].
Feature-based approach: This approach extracts important features from labeled images and utilizes these to recommend informative images from the unlabeled dataset. The authors of [
14] proposed
Similarity-based Active Learning (SAL) to handle real-world scenarios with both a significant class imbalance and a limited labeled dataset. During training, SAL creates many combinations of triplets, each with two images of the same class and one image of a different class. SAL optimizes the similarity-based triplet loss to generate a feature extractor that best discriminates between classes. SAL calculates one class-feature representation by averaging the features of all the labeled images of that class. As a result, the rare features of the class are excluded from the class-feature representation. SAL recommends unlabeled images for each class based on the distance between the class-feature representation and the features of the unlabeled images. Like CEAL, SAL also pseudo-labels unlabeled images with a high classification probability given a threshold and automatically adjusts the threshold value for pseudo-labeling for each iteration.
The authors of the Core-Set active learning method [
27] proposed a new core-set loss, which is the difference between the average empirical loss over labeled data points and that over the entire dataset including the unlabeled data. For each iteration, solving the
k-Center problem [
34] on the unlabeled data points minimizes the upper bound of the loss. The unlabeled images at the centers are recommended for each iteration. This method is computationally expensive, especially with a large unlabeled dataset since it requires building a large distance matrix to solve the
k-Center problem [
3]. Related to Core-Set, the authors of ProbCover [
35] proposed an active learning method for a low-budget regime that maximizes probability coverage.
Variational autoencoder approach: This approach utilizes
Variational Autoencoder (VAE) to learn the latent space of labeled images. Then, the learned representations are used to recommend the most informative images to be labeled by a human expert. The authors of
Variational Adversarial Active Learning (VAAL) [
36] adapted VAE to generate image representations similar to those of labeled and unlabeled images. VAAL uses an adversarial network that learns to discriminate the feature representations of the unlabeled and labeled images. The goal is to recommend unlabeled images with features different from those found in the labeled dataset. VAAL is task-agnostic.
Task-Aware Variational Adversarial Active Learning (TA-VAAL) [
37] alters VAAL to be task aware by considering the classification probability of each unlabeled data point before labeling. TA-VAAL uses the loss ranking prediction [
30] and embeds the normalized ranking loss information from the image classifier to identify images with low classification probability.
Variational Bayes for Active Learning (VaB-AL) [
23] uses VAE to estimate the lower bound of the likelihood of misclassifying an unlabeled image. VaB-AL was shown to perform better than the Core Set algorithm [
27], VAAL, and DBAL [
31]. Like TA-VAAL, VaB-AL is task aware and performs well under class imbalance.
Compared to recent DAL methods that work well for class imbalance based on VAE, such as TA-VAAL and VaB-AL, the proposed VisActive works well with even more limited labeled data. VisActive does not rely on the training of the auto-encoder and auto-decoder, which usually requires comparably more training data. Furthermore, VisActive gives fine-grained control over its recommendation, e.g., finding rare concepts, cross-concepts, and unseen concept sets of images. Therefore, the final image classifier is effective across different image characteristics of rare and common classes. Compared to SAL, VisActive gives more fine-grained features at the object or part of object levels. Hence, VisActive can be more intentional about what to recommend.
2.2 Visual Concept Layer
Khaleel et al. previously proposed the
visual concept (VC) layer to learn visual concepts automatically and use them to explain a classifier's decision on a given image [
21]. In VisActive, we utilize this component to learn different key feature patterns in the training data. We summarize the VC layer here. Let
\(fm\) represent
\(d\) feature maps of size
\(W \times H\) values;
\(fm\) is the output of the last convolutional layer of the classifier of a given image. Each neuron in the VC layer learns one visual concept for a given class using a filter
\(vc \in {\mathbb{R}}^{( {h + 1} ) \times ( {w + 1} ) \times d}\) , where
\(1 \le h < H\) and
\(1 \le w < W\) are the predefined height and width of the visual concept, respectively. Choosing appropriate values for
\(h\) and
\(w\) depends on the input image size and the depth of the convolutional network. For example, for a 64
\(\times\) 64 image, we recommend setting the value of
\(h\) and
\(w\) to four with shallow networks and to one or two for a deeper network. The visual concept size corresponds to a receptive field at the input layer. We recommend choosing a visual concept size so that its receptive field is large enough to contain most of an object for classification. The filter
\(vc\) is randomly initialized. The training process keeps adjusting the filter values to reduce the training loss. The resulting values of the filter
\(vc\) represent one visual concept for its class. The loss function used in [
21] relies on the distance function in Equation (
1) between the filter
\(vc\) and the corresponding feature maps partition
\(fv \in {\mathbb{R}}^{( {h + 1} ){\rm{\ }} \times {\rm{\ }}( {w + 1} ){\rm{\ }} \times d}\) when sliding the filter
\(vc\) across
\(fm\) with a stride of one:
As shown in Figure
2, the function
\(\|vc - \lambda ( {fv} )\|_2^2\) calculates the squared
\({L}_2\) distance for each of the
\(d\) dimension vectors of
\(vc\) and
\(\lambda ( {fv} )\) . The
\(\lambda ( {fv} )\) multiplies each border cell in
\(fv\) with an importance constant between 0 and 1. The use of the
\(\lambda ( {fv} )\) function keeps the most important features of the target concept in the central part of the filter. Next,
\(avgpool()\) , with the window size
\(h\ \times w\) and a stride of one, outputs the average of the values under the window, which gives a matrix of
\(h\ \times w\) values. Then,
\(minpool()\) with window size
\(h\ \times w\) returns the minimum distance score between
\(vc\) and
\(fv\) . Sliding
\(vc\) across
\(fm\) results in a distance matrix
\(D{M}^{vc}\) that stores the minimum average distance between
\(vc\) and each partition
\(fv\) of
\(fm\) . Last, we select the minimum distance in
\(D{M}^{vc}\) using the
\(min( \ )\) function and convert the result to a similarity score, as shown in Equation (
2). The result of Equation (
2) is the highest similarity score of the visual concept
\(vc\) in the feature map
\(fm\) . The value range of
\(sim( {vc,\ fm} )\) is between 0 and 1. We utilize the VC layer in our proposed method.