Research on image classification method based on improved multi-scale relational network

Omniglot data set

In 2011, the Omniglot data set was collected by Brenden Lake and his collaborators at MIT through Amazon’s Mechanical Turk (Lake et al., 2011; Zheng et al., 2015). It consists of 50 international languages, including mature international languages such as Latin and Korean, little-known local dialects, and fictional character sets such as Aurek-Besh and Klingon. The number of letters in each language varies widely, from about 15 to 40 letters, with 20 samples for each letter. Therefore, the Omniglot data set consists of 1623 categories and 32,460 images. Figure 1 illustrates the five languages of the Omniglot data set.

MiniImageNet data set

MiniImageNet data set, proposed by Vinyals et al., consists of 60,000 84 × 84 ×3 color images, a total of 100 categories, each with 600 samples (Vinyals et al., 2016). The distribution of data sets is very different, and the image category involves animals, household goods, remote sensing images, food, etc. Vinyals did not publish this data set. Ravi and Larochelle randomly selected 100 classes from the ImageNet data set to create a new MiniImageNet data set, which was divided into training set, validation set and test set at a ratio of 64:16:20 (Ravi & Larochelle, 2016).

Metalearning based on metric Learning

In meta-learning, metric learning refers to learning similarity measurement from a wide range of task Spaces, so that the experience extracted from the previous learning tasks can be used to guide the learning of new tasks to achieve the purpose of learning how to learn. The learner (meta-learner) learns the target set on each task of the training set by measuring the distance of the support set, and finally learns a metric. Then for the new task of the test set, it can quickly classify the target set correctly with the help of a small number of samples of the support set.

At present, the methods of small sample image classification based on metric Learning include: twin network, matching network, prototype network and relational network. The twin network is composed of two identical convolutional neural networks, and the similarity between two images is calculated by comparing the loss function with the input of paired samples. The other three methods, instead of using paired sample inputs, calculate the similarity between the two images by setting the support set and the target set. In this paper, the improvement is made on the basis of multi-scale relational network, so the main structure of this method is mainly introduced below.

The structure of relational network (Sung et al., 2018) first obtains feature graphs of support set and target set samples through embedded modules, then splices feature graphs of support set and target set samples in depth direction, and finally obtains relationship score by learning splicer features through relationship module, so as to determine whether support set and target set samples belong to the same category.

The calculation formula of relational fractions is as follows: (1)${\displaystyle {\mathrm{r}}_{i,j}=g_{\varphi }\left(C\left(f_{\phi }\left(x_i\right),f_{\phi }\left(x_j\right)\right)\right),i=1,2,\dots ,C}$

Where, x_i represents the supporting set sample, x_j represents the target set sample, f_φ(x) represents the embedded module, f_φ(x_i) and f_φ(x_j) represent the feature graph of the supporting set and target set sample, and $C\left(f_{\phi }\left(x_i\right),f_{\phi }\left(x_j\right)\right)$ represents the feature graph splicing operator. g_ϕ stands for relationship module.

Model-independent meta-learning algorithm

According to the idea of transfer learning, when adapting to a new task, MAML only needs to fine-tune the learner so that the parameter of the learner is adapted from θ to ${\theta }_i^{\prime }$. However, different from migration learning, MAML only needs to perform one or more gradient descent iterative steps on a small amount of data in new task T_i to enable the learner to converge to the optimal parameter.

Here, ${\theta }_i^{\prime }$ only considers the MAML that converges to the optimal parameter after an iterative step of gradient descent as follows: (2)${\displaystyle {\theta }_i^{\prime }=\theta -\alpha {\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)}$

Where, $f\left(\theta \right)$ stands for learner, L_Ti stands for loss on specific task, ${\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)$ stands for loss gradient, and α stands for learner gradient update pace, namely learning rate of learner.

On a small amount of data in new task T_i, the learner can be converged to the optimal parameter ${\theta }_i^{\prime }$ after one step of gradient descent iteration, so MAML must find a set of initialization representations that can be effectively fine-tuned according to a small number of samples. In order to achieve this goal, based on the idea of meta-learning “using previous experience to quickly learn new tasks”, learners need to learn the learner parameter θ on different tasks. MAML defines this process as meta-learning process.

In the meta-learning process, for different training tasks, the optimal parameter ${\theta }_i^{\prime }$ suitable for the specific task was obtained through a step of gradient iteration, and then sampled again on each task for testing, requiring $L_{T_i}f\left({\theta }_i^{\prime }\right)$ to reach the minimum value. Therefore, MAML adopted the sum of test errors on different tasks as the optimization objective of the meta-learning process, as shown below: (3)${\displaystyle mi{n}_{\theta }\sum _{T_i\sim p\left(T\right)}{L}_{T_i}f\left({\theta }_i^{\prime }\right)=\sum _{T_i\sim p\left(T\right)}{L}_{T_i}f\left(\theta -\alpha {\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)\right)}$

where, $p\left(T\right)$ represents the distribution of the task set, L_Ti represents the loss on the specific task, ${\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)$ represents the loss gradient, and α represents the learning rate of the learner.

It can be seen from Eq. (3)–(3) that the optimization goal of the meta-learning process is to adopt the updated model parameter ${\theta }_i^{\prime }$ adapted to the specific task, while the meta-optimization process is ultimately executed on the learner parameters θ. The stochastic gradient descent method is adopted, and the updated iteration formula of model parameter θ is shown as follows: (4)${\displaystyle \theta \leftarrow \theta -\beta {\nabla }_{\theta }\sum _{T_i\sim p\left(T\right)}{L}_{T_i}f\left({\theta }_i^{\prime }\right)}$

where, $p\left(T\right)$ represents the distribution of the task set, L_Ti represents the loss on the specific task, ${\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)$ represents the gradient, and β represents the gradient update pace of the meta-learner, also known as the learning rate of the meta-learner.

Substituting Eq. (3)–(1) into Eq. (3)–(4), we can get: (5)${\displaystyle \theta \leftarrow \theta -\beta {\nabla }_{\theta }\sum _{T_i\sim p\left(T\right)}{L}_{T_i}f\left(\theta -\alpha {\nabla }_{\theta }{L}_{T_i}f\left(\theta \right)\right)}$

Algorithm design of multi-scale meta-relational network

In the multi-scale meta-relational network, we hope to find a set of characterization θ that can make fine adjustments efficiently according to a small number of samples. Where, θ is composed of feature extractor parameter φ and metric learner parameter ϕ. During the training process, each task is composed of training set D_train and test set D_test. Where, D_train is used for the inner optimization iteration, D_test serves as the target set of the outer optimization iteration, and the support set adopts the support set D_S in D_train.

In the inner optimization iteration process, on the new task T_i with a small amount of data D_train, the learner converges to the optimal parameter θ_i after an iterative step of gradient descent, as shown below: (6)${\displaystyle {\theta }_i=\theta -\alpha {\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)}$

Where α is a vector of the same size as $L_{T_i,D_{train}}^{\prime }\left(\theta \right)$, representing the learning rate of the learner. The direction of the vector represents the update direction, and the magnitude of the vector represents the learning step.

For different training tasks, the optimal parameter θ_i suitable for specific tasks was obtained through a step of gradient iteration, and then tested on D_test of different tasks, requiring $L_{T_i,D_{test},D_S}f\left({\theta }_i\right)$ to reach the minimum value. Therefore, the meta-optimization objectives of the meta-training process are as follows: (7)${\displaystyle mi{n}_{\theta }\sum _{T_i\sim p\left(T\right)}{L}_{T_i,D_s,D_{test}}f\left({\theta }_i\right)}$

Substituting Eq. (3)–(6) into Eq. (3)–(7), we can find: (8)${\displaystyle mi{n}_{\theta }\sum _{T_i\sim p\left(T\right)}{L}_{T_i,D_S,D_{test}}f\left(\theta -\alpha {\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\right)}$

For parameter θ, the gradient calculation formula of formula (3–8) is as follows: (9)${\displaystyle g_{\theta }=\frac{\partial }{\partial \theta }{L}_{T_i,D_S,D_{test}}f\left(\theta -\alpha {\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\right)}$

Substituting Eq. (3)–(6) into Eq. (3)–(9) can be simplified as follows: (10)${\displaystyle \begin{array}{cc}{g}_{\theta }\hfill & =\frac{\partial L_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}{\partial f\left({\theta }_i\right)}\times \frac{\partial f\left({\theta }_i\right)}{\partial {\theta }_i}\times \frac{\partial {\theta }_i}{\partial \theta }\hfill \\ \hfill & =\frac{\partial L_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}{\partial f\left({\theta }_i\right)}\times \frac{\partial f\left({\theta }_i\right)}{\partial {\theta }_i}\times \left(I-\alpha \frac{\partial \left({\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\right)}{\partial \theta }\right)\hfill \end{array}}$

where I is the unit vector.

It can be seen from Eq. (3)–(10) that the update process of the multi-scale meta-relational network involves the gradient calculation process, that is, when the gradient operator in the meta-target is used to propagate the meta-gradient, additional reverse transfer is required to calculate the Hessian vector product (Kemp & Maas, 2009).

In the multi-scale relational network, except for the sigMIod nonlinear function at the last full connection layer of the metric learner, all other nonlinear functions are ReLU. The ReLU neural network is almost linear in part, which indicates that the second derivative is close to zero in most cases. Therefore, the multi-scale element relational network, like MAML, ignores the second derivative in the calculation of the back propagation of the element gradient (Kemp & Maas, 2009), as follows: (11)${\displaystyle \begin{array}{cc}{g}_{\theta }\hfill & =\frac{\partial L_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}{\partial f\left({\theta }_i\right)}\times \frac{\partial f\left({\theta }_i\right)}{\partial {\theta }_i}\hfill \\ \hfill & =\frac{\partial L_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}{\partial {\theta }_i}\hfill \\ \hfill & ={\nabla }_{{\theta }_i}{L}_{T_i,D_S,D_{test}}f\left({\theta }_i\right)\hfill \end{array}}$

Therefore, in the process of calculating the outer gradient, the multi-scale element relation network stops the back propagation after calculating the gradient at θ_i and calculates the second derivative at θ.

For parameter α, the gradient calculation formula of formula (3–8) is as follows: (12)${\displaystyle \begin{array}{cc}{g}_{\alpha }\hfill & =\frac{\partial }{\partial \alpha }{L}_{T_i,D_S,D_{test}}f\left(\theta -\alpha {\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\right)\hfill \\ \hfill & =\frac{\partial L_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}{\partial {\theta }_i}\times \frac{\partial {\theta }_i}{\partial \alpha }\hfill \\ \hfill & =-{\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\times {\nabla }_{{\theta }_i}{L}_{T_i,D_S,D_{test}}f\left({\theta }_i\right)\hfill \end{array}}$

In the outer optimization iteration process, stochastic gradient descent method is adopted, and the updated iteration formula of model parameter θ is shown as follows: (13)${\displaystyle \theta \leftarrow \theta -\beta \sum _{T_i\sim p\left(T\right)}{\nabla }_{{\theta }_i}{L}_{T_i,D_S,D_{test}}f\left({\theta }_i\right)}$

Similarly, the update iteration formula of model parameter α is shown as follows: (14)${\displaystyle \alpha \leftarrow \alpha +\beta \sum _{T_i\sim p\left(T\right)}\left({\nabla }_{{\theta }_i}{L}_{T_i,D_S,D_{test}}f\left({\theta }_i\right)\right)\left({\nabla }_{\theta }{L}_{T_i,D_{train}}f\left(\theta \right)\right)}$

In each inner iteration, all tasks can be used to update the gradient, but when the number of tasks is very large, all tasks need to be calculated in each iteration step, so the training process will be slow and the memory capacity of the computer will be high. If each inner iteration uses one task to update the parameters, the training speed will be accelerated, but the experimental performance will be reduced, and it is not easy to implement in parallel. So, during each inner iteration, we take num_inner_task for the number of tasks. By selecting the number of tasks in the inner iteration process as a super parameter, memory can be reasonably utilized to improve the training speed and accuracy.

During the meta-training of multi-scale meta-relational network, after the end of an epoch, the accuracy on the meta-validation set was calculated, and the highest accuracy on the meta-validation set was recorded so far. If consecutively N epoches (or more, adjusted according to the specific experimental conditions) did not reach the optimal value, it could be considered that the accuracy was no longer improved, and the iteration could be stopped when the accuracy was no longer improved or declined gradually. Output the model with the highest accuracy, and then test it with this model.

Omniglot data set

In the Omniglot data set experiment, all samples were processed into a size of 28 ×28 and a random rotation was used to enhance the data. From 1,623 classes, 1200, 211 and 212 classes were selected as meta-training set, meta-validation set and meta-test set respectively. In this section, single-sample image classification experiments of 5-ways 1-shot and 20-ways 1-shot and small-sample image classification experiments of 5-ways 5-shot and 20-ways 5-shot are conducted.

In the experiment, 100 episodes of multi-scale meta-relational network were taken as one epoch. The iteration times of the meta-training set, meta-validation set and meta-test set are accordingly 70,000, 500 and 500. Other super-parameter selections on the Omniglot dataset are shown in Table 1.

Single sample image classification

It can be seen from Fig. 2 that, when the number of iterations is 42400, the multi-scale meta-relational network achieves the highest accuracy rate of 99.8667% in the 5-way 1-shot experiment on the meta-validation set. The iteration time of the multi-scale meta-relational network is less than that of the multi-scale relational network (method 1 in this paper). Therefore, compared with the convergence of the multi-scale relational network when the iteration reaches 114,000, the learning speed of the multi-scale meta-relational network is faster than that of the multi-scale relational network.

The trained model was tested on the meta-test set, and the test results were shown in Table 2. According to Table 2:

1. The accuracy rate of the 5-way 1-shot experiment on the meta-test set of the multi-scale meta-relational network is higher than that of MAML and Meta-SGD, and about 0.22% higher than that of the multi-scale meta-relational network.

2. MAML and Meta-SGD based on optimized initial characterization need fine-tuning on new tasks, while the multi-scale relational network based on metric Learning can achieve good generalization performance on new tasks without fine-tuning. By comparing the two methods, the accuracy rate of the 5-way 1-shot experiment on the meta-test set of the multi-scale relational network is higher than MAML, but slightly lower than meta-SGD.

As can be seen from Fig. 3, when the number of iterations is 57,700, the multi-scale meta-relational network (method 2 in this paper) achieves the highest accuracy rate of 98.86% in the 20-way 1-shot experiment on the meta-validation set. Compared with the multi-scale relational network (method 1 in this paper), the learning speed of multi-scale meta-relational network is faster than that of multi-scale relational network when it iterates to 89,000.

The trained model was tested on the meta-test set, and the results were shown in Table 2 below. According to Table 2:

1. The accuracy rate of the 20-way 1-shot experiment of the multi-scale meta-relational network on the meta-test set is about 0.47% higher than that of the multi-scale meta-relational network, which is higher than MAML and Meta-SGD.

2. MAML and Meta-SGD based on optimal initialization representation were compared with the multi-scale relational network based on metric Learning. The accuracy of the multi-scale relational network in the 20-way 1-shot experiment on the meta-test set was higher than that of MAML and meta-SGD.

Single sample image classification

As can be seen from Fig. 6, with the increase of iteration, loss decreases to convergence, and accuracy gradually increases to convergence. When the number of iterations is 87,000, the multi-scale meta-relational network (method 2 in this paper) achieves the highest accuracy rate of 51.234% in the 5-way 1-shot experiment on the meta-validation set. The iteration time of multi-scale meta-relational network is less than that of multi-scale relational network (method 1 in this paper). Therefore, compared with the convergence of multi-scale relational network when iteration reaches 155,000, the learning speed of multi-scale meta-relational network is faster than that of multi-scale relational network.

The trained model is tested on the meta-test set, and the results are shown in Table 5. According to Table 5:

1. The accuracy rate of the 5-way 1-shot experiment on the meta-test set of the multi-scale meta-relational network is about 0.35% higher than that of the multi-scale meta-relational network, and higher than MAML and Meta-SGD.

2. MAML and Meta-SGD based on optimized initial characterization were compared with the multi-scale relational network based on metric learning. The accuracy of the 5-way 1-shot experiment on the meta-test set of the multi-scale relational network was higher than MAML, but lower than meta-SGD.

Small sample image classification

As can be seen from Fig. 7, with the increase of iteration, loss decreases to convergence, and accuracy gradually increases to convergence. When the number of iterations is 62000, the multi-scale meta-relational network (method 2 in this paper) achieves the highest accuracy rate of 66.81% in the 5-way 5-shot experiment on the meta-validation set. Compared with the convergence of multi-scale relational network (method 1 in this paper) when iterative to 140,000, the learning speed of multi-scale meta-relational network is faster than that of multi-scale relational network.

The trained model is tested on the meta-test set, and the results are shown in Table 6. According to Table 6:

1. The accuracy rate of the 5-way 5-shot experiment on the meta-test set of the multi-scale meta-relational network is about 0.34% higher than that of the multi-scale meta-relational network, which is higher than MAML and Meta-SGD.

2. MAML and Meta-SGD based on optimized initial characterization were compared with the multi-scale relational network based on metric learning. The accuracy of the multi-scale relational network in the 5-way 5-shot experiment on the meta-test set was higher than that of MAML and meta-SGD.

In summary, the experimental results on Miniimagenet data set are as follows:

1. The classification accuracy of MNS on Miniimagenet data set is higher than that of MNS, MAML and META-SGD, and the training speed is higher than that of MNS.

2. By comparing the MAML and Meta-SGD based on optimal initial representation and the multi-scale relational network based on metric learning, the classification accuracy of the 5-way 1-shot experiment on Miniimagenet data set was slightly lower than that of meta-SGD, and the classification accuracy of the 5-way 5-shot experiment was higher than that of the META-SGD and MAML.