GrOD: Deep Learning with Gradients Orthogonal Decomposition for Knowledge Transfer, Distillation, and Adversarial Training

While one can fix the over-regularization issue through lowering the regularizer weight for statistical learning, such problem is still tough for regularized deep learning [17]. For example, using the starting point as the reference (SPAR) (i.e., incorporating an L \( ^2 \) -norm regularizer that constrains the distance between the parameter and the starting point of optimization [2]) is frequently used to fine-tune deep neural networks for deep transfer learning. Using an inappropriate pre-trained model for SPAR leads to even worse performance than the one from scratch [24, 25], as the L \( ^2 \) -norm regularization with the start point of optimization would affect the local minimum points that the learning procedure finally converges to, while the selection of poor local minimum points may significantly hurt the generalization performance of deep learning.

We specify above observation using an example based on \( L^2 \) -SP [2] shown in Figure 1. The Black Line refers to the empirical loss descent flow of common gradient-based learning algorithms with pre-trained weights as the start point. It shows that with the empirical loss gradients as the descent direction, such method quickly converges to a local minimum in a narrow cone, which is usually considered as an over-fitting solution. In the meanwhile, the Blue Line demonstrates the possible empirical loss descending path of \( L^2 \) -SP algorithm, where a strong regularization blocks the learning algorithm to continue lowering the empirical loss while traversing the area around the point of pre-trained weights. An ideal case has been illustrated as the Red Line, where \( L^2 \) -SP regularizer helps the learning algorithm to avoid the over-fitting solutions. The overall descent direction adapting \( L^2 \) -SP regularizer with respect to empirical loss leads to generalizable solutions. There thus needs a method to make both empirical loss and regularizer continue descending to boost the performance of deep transfer learning. Thus, the new technique to balance the regularization term and the empirical loss in the deep learning procedure is needed.

Motivated by above observations, simple yet effective training paradigm, namely Gradients Orthogonal Decomposition (GrOD), which provides a new descent direction estimator for the regularized learning of over-parameterized deep neural networks. GrOD follows a simple Empirical Risk Minimization(ERM) Preserved descent direction principle—in every iteration of the learning procedure, the empirical loss of regularized deep learning should descend as fast as the one based on ERM. Specifically, GrOD decomposes the gradients of the regularizer term and removes the part that is opposed to the empirical loss gradients. With remaining parts combined, the regularizer and empirical loss terms are expected to be “minimized simultaneously” while the minimization of empirical loss is more preferred in GrOD.

Inspired by the observation that regularizer may hurt the model’s fitting by preventing empirical loss descending, we proposed a novel regularized deep learning framework GrOD that improves regularized deep learning with a better balance between the gradients of loss function and the regularization term—i.e., with better fitness to the training data without simply degrading the weight of regularization term. During the learning procedure, when the angle between the empirical loss gradient and the regularizer gradient is large (larger than \( 90^\circ \) ), GrOD decomposes the regularizer gradients into two components: hurting part (parallel to empirical loss gradients) and safely regularized part (vertical to empirical loss gradients), discards the hurting part and preserves the safely regularized part. In this way, it will preserve regularization affects without preventing empirical loss descending.

In terms of methodology, the most relevant work to our study is Gradient Episodic Memory (GEM) for continual learning [26], which continuously learn the new task using the well-trained models for past tasks. In terms of objectives, GrOD aims at lowering the effects of regularization from hurting ERM, while GEM prevents the ERM from hurting regularization effects (i.e., the accuracy on old tasks). In terms of algorithms, in every iteration of learning, GEM estimates the descent direction with respect to the gradients of the new task and all past tasks using a time-consuming Quadratic Program (QP), while GrOD re-estimates the descent direction from the gradients of the regularizer term and the empirical loss term with low-complexity orthogonal decomposition. All in all, GEM can be considered as a special case of GrOD using L \( ^2 \) -SP regularizer [2] based on two tasks.

Extensive experiments have been performed using state-of-the-art algorithms for deep transfer learning [2], knowledge distillation [3], and adversarial learning [4] tasks, on top of a wide range of deep learning benchmark datasets including Caltech, MIT indoor 67, CIFAR-10, Fashion MNIST, and ImageNet. The experiments show that GrOD always improves the performance of the three tasks. Specifically, for transfer learning tasks, GrOD has improved the \( L^2 \) -SP algorithm [2] with 0.1%–7% higher accuracy (even transferring from the network pre-trained by inappropriate datasets). Through knowledge distillation [3], the network trained by GrOD outperforms the vanilla one with 0.5%–5% higher accuracy through aligning the generated feature maps. For adversarial learning task, GrOD has been evaluated to enhance the state-of-the-art algorithm [4] with significant Pareto-improvement in both accuracy and robustness. Besides, the gradients direction analysis based on the experiments verified our assumptions about the descent direction’s performance during neural network’s training process.

Deep convolutional networks usually consist of a great number of parameters that need fit to the dataset. For example, ResNet-110 has more than one million free parameters. The size of free parameters causes the risk of over-fitting. Regularized deep learning is the technique to reduce this risk by constraining the parameters within a limited space with respect to a set of regularization terms. The general learning problem is usually formulated as follow.

When the training dataset size is relatively small, we often need to transfer knowledge learned from large datasets to small tasks [27, 28, 29, 30, 31]. Given the weights of a deep neural network pre-trained by a large dataset (e.g., ImageNet), a recent work [2] proposed to first use pre-trained weights as the starting point of the training procedure, then leverages the squared Euclid distance from the training weights to the pre-trained weights as the regularization term for deep transfer learning. Such approach “helps” the training procedure find a generalizable solution with higher accuracy, even based on a small set.

In terms of regularization, given the weights (denoted as \( \mathbf {\Omega }_s \) ) of a neural network pre-trained from a large dataset, \( L^2 \) -SP [2] algorithm uses the squared-euclidean distance from \( \mathbf {\omega } \) to the pre-trained weights \( \mathbf {\omega _s} \) of source network (listed in Equation (2)) to constrain the learning procedure whereIn terms of optimization procedure, \( L^2 \) -SP makes the learning procedure start from the pre-trained weights (i.e., using \( \mathbf {\omega _s} \) to initialize the learning procedure).

In addition to above regularization, other methods have been used for deep transfer learning, including [19, 32, 33, 34, 35, 36]. As early as in 2014, authors in [32] reported their observation of significant performance improvement through directly reusing weights of the pre-trained source network to the target task, when training a large CNN with tremendous number of filters and parameters. However, in the meanwhile of reusing all pre-trained weights, the target network might be overloaded by learning tons of inappropriate features (that cannot be used for classification in the target task), while the key features of the target task have been probably ignored. In this way, Yosinki et al. [37] proposed to understand whether a feature can be transferred to the target network, through quantifying the “transferability” of features from each layer considering the performance gain. Furthermore, Huh et al. [19] made empirical study on analyzing features that CNN learned from ImageNet dataset to other computer vision tasks, so as to detail the factors effecting deep transfer learning accuracy. In recent days, this line of research has been further developed with increasing number of algorithms and tools that can improve the performance of deep transfer learning, including subset selection [33, 38], sparse transfer [34], filter distribution constraining [35], parameter transfer [36], and transfer learning over manifolds [39]. Moreover, [29] studies the memorability of images using transfer learning, while authors in [30, 40] work on the knowledge transfer crossing the modalities. The overall survey on transfer learning can be found in [25, 41].

To achieve similar goals, instead of adopting the weights in a straightforward approach, authors [3] propose to use so-called knowledge distillation mechanism, where given a pre-trained network as the teacher network it considers the training objective as a student network that learns from the teacher. More specific, the squared Euclid distance between feature maps generated by the convolutional layers of the teacher and student networks are used as regularization [22]. The feature-wise knowledge distillation algorithm proposed in [3] enables effective knowledge transfer through learning the behaviors of the pre-trained network, as a gift of knowledge distillation. Similar mechanism is also used for neural network compression [42], using the original network as the teacher and the compression target model as the student with feature map quantization.

Given the training dataset \( \lbrace (\mathbf {x}_1,y_1),\ldots ,(\mathbf {x}_n,y_n)\rbrace \) and N filters in the teacher/student networks for knowledge distillation, the knowledge distillation algorithm [3] models the regularization as the aggregation of squared-euclidean distances between feature maps outputted by the N filters of the teacher/student networks, such thatwhere \( \mathbf {F}_j(\mathbf {\omega },\mathbf {x}_i)) \) refers to the feature map outputted by the jth filter ( \( 1\le j\le N \) ) of the target network based on weight \( \mathbf {w} \) using input image \( \mathbf {x}_i\ \) ( \( 1\le i\le n \) ).

In terms of methodologies, the knowledge distillation was originally proposed to compress deep neural networks [22, 31, 43] through teacher–student network training, where the teacher and student networks are usually based on the same task [22]. In terms of inductive transfer learning, authors in [3] were first to investigate the possibility of using the distance of intermediate results (e.g., feature maps generated by the same layers) of source and target networks as the regularization term. Furthermore, [44] proposed to use the distance between activation maps as the regularization term for so-called “attention transfer”. Notice in our experiment settings, we mainly focus on the applications of knowledge distillation to knowledge transfer, i.e., the source model is pre-trained using other datasets.

In addition to the accuracy improvement, there frequently needs to enhance the robustness of deep learning under adversarial attacks [45, 46]. With a strategy to perturb the training data for adversarial samples generation, [46] proposed to incorporate the training loss based on the generated adversarial samples via Fast Gradient Sign Method (FGSM) as the regularization term to augment the loss for deep adversarial learning. Reference [4] indicated that instead of FGSM, using adversarial examples generated by Projected Gradient Descent (PGD, [47]) on the negative loss function will obtain a more robust model.

Given the training dataset \( \lbrace (\mathbf {x}_1,y_1),\ldots ,(\mathbf {x}_n,y_n)\rbrace \) , one state-of-the-art adversarial learning algorithm [4], studied in this article, first synthesizes the adversarial samples set \( \lbrace (\mathbf {x}^{\prime }_1,y_1),\ldots ,(\mathbf {x}^{\prime }_n,y_n)\rbrace \) , through perturbation. Then, the algorithm proposes to use the empirical loss based on adversarial samples as the objective function to minimize whereUsing first-order taylor expansion, we can approximate (4) as the regularized formThus, the regularization part of adversarial training is \( \Omega (\omega) = \tfrac{1}{n}\sum _{i=1}^n (\mathbf {x}^{\prime }_{i} -\mathbf {x}_{i})\tfrac{\partial }{\partial x}L(z(\mathbf {x}_{i},\mathbf {\omega }), y_{i}) \) with \( \lambda =1 \) . Specifically, a pre-trained model using the original dataset is frequently required as the target network for defense, where the gradients and/or Hessian matrices of the loss function are used to perturb the input space of the training data with optional noise to the labels. One can also generate the perturbations for adversarial learning under black–box/derivative–free settings [48, 49]. In addition to the empirical loss over the perturbed set, knowledge distillation over feature maps can also be adopted for defense [50, 51]. More definitions and details could be found in a comprehensive survey [52].

Compared to above work and other transfer learning studies, our work aims at providing a generic descent direction estimation strategy that improves the performance of regularization-based deep transfer learning. The intuition of GrOD is, per iteration during the learning procedure, re-estimating a new direction of loss descending that addresses the affect of regularizers while making the empirical loss reduction/minimization not hurt. In our work, we demonstrated the capacity of GrOD working with two most recent deep transfer learning regularizers— \( L^2 \) -SP [2] and Knowledge distillation [3], which are based on two typical deep learning philosophies (i.e., constraining weights and feature maps, respectively), using a wide range of transfer learning tasks. The consistent performance boosts with GrOD in all cases of experiments suggests that GrOD can improve above regularization-based deep transfer learning with higher accuracy.

Other techniques, including continual learning [20, 21], attention mechanism for CNN models [44, 53, 54, 55], and so on, can also improve the performance of knowledge transfer between tasks. We believe our work made complementary contributions in this area. All in all, we appreciate the contributions made by these studies. Furthermore, compared to the earlier version of this manuscript [1], we have made significant contributions to extend the previous work that primarily focuses on deep transfer learning, to improve the regularized deep learning in general cases. New regularized deep learning applications, such as knowledge distillation and adversarial learning, have been studied here. This manuscript includes our most recent efforts on improving deep transfer learning, adversarial learning, and network distilling with GrOD, from both theoretical and empirical aspects. Additional experiments with new results have been provided to demonstrate our new findings.

Prior to presenting of the algorithms, this section introduces the settings of the problem.

Following the above definition, we reduce our research problem as finding a new descent direction based on the gradients of the empirical loss and the regularizer term. The new descent direction is expected to preserve the effects of regularization, while avoiding to hurt the empirical loss minimization. Due to the affect of regularization \( \Omega (\mathbf {\omega },\mathbf {\omega }_s) \) , the angle between the actual descent direction \( \mathbf {d}(\mathbf {\omega }) \) and the gradient of empirical loss \( \nabla J(\mathbf {\omega }) \) , i.e., \( \measuredangle (\mathbf {d}(\mathbf {\omega }),\nabla J(\mathbf {\omega })) \) , would be large. It is intuitive to state that when \( \measuredangle (\mathbf {d}(\mathbf {\omega }),\nabla J(\mathbf {\omega }) \) is large, the descent direction cannot effectively lower the empirical loss and causes the potential performance bottleneck of deep transfer learning. We thus formulate the technical problem with following assumptions specified.

Based on above definitions and assumptions, in our research, we propose a new direction descent algorithm—every iteration of the algorithm re-estimates a new descent direction \( \mathbf {\widehat{d}} \) to effectively lower the training loss based on the optimization object \( \mathbf {\omega } \) while preserving the effect of regularizer \( \Omega (\mathbf {\omega }) \) (Assumption 2). Note that, to avoid the use of any threshold for bounding the two angles between \( \mathbf {\widehat{d}} \) and \( \nabla J(\omega) \) and between \( \mathbf {\widehat{d}} \) and \( \nabla Omega(\omega) \) , we formulate the research problem as follows.

In this section, we presented the design of GrOD as a descent direction estimator that solves the relaxed constraint programming problem addressed in Section 3.1.2. Given the empirical loss function \( J(\omega) \) , the regularization term \( \Omega (\mathbf {\omega }) \) , the set of training data \( \mathbf {D}=\lbrace (\mathbf {x}_1,y_1),(\mathbf {x}_2,y_2),\ldots ,(\mathbf {x}_n,y_n)\rbrace \) , the mini-batch size b and the regularization coefficient \( \lambda \) , we propose to use Algorithm 1 to estimate the descent direction at the point \( \omega _t \) for the tth iteration of regularized deep learning.

With such descent direction estimator, the learning algorithm is capable of replacing the original stochastic gradient estimators used in stochastic gradient descent (SGD), Momentum and/or Adam for deep learning. Specifically, for each (e.g., the tth) iteration of learning procedure, GrOD estimates the gradients of empirical loss and regularization term (i.e., \( \nabla \widehat{J}_t \) and \( \nabla \widehat{\Omega }_t \) ) separately as follows.

Note that the complexity of GrOD for descent direction estimation is low. Given the two gradient vectors \( \nabla \widehat{J}_t \) and \( \nabla \widehat{\Omega }_t \) , GrOD uses Line 10 of Algorithm 1 to estimate the descent direction, where the inner-product of two vectors, scalar-vector-product, and some vector addition/subtraction operations are used. Thus the computational complexity of Line 10 of GrOD for descent direction estimation is \( O(d) \) , where d refers to the number of dimensions.

To understand the theoretical properties of GrOD, please refer to Section 3.4 for our analysis. For empirical validation, please refer to Section 4.5.2, where the experiment results validate the effect of the gradient orthogonal decomposition to the regularized deep learning.

Based on the algorithm introduced in Algorithm 1, we can make lemmas as follow.

Above lemma could be obtained using the proof as follow.

Finally, we could obtain our theoretical result as follow.

In this way, we could conclude GrOD is the solution that we desire in problem (8). Furthermore, from the perspectives of descent directions, we also find that the behavior of GrOD is not achievable through tuning the weight of the regularizer alone.

To interpret the theoretical results, we use an example to visualize our intuition. Figure 2 illustrates an example of GrOD descent direction estimation, when the angles between gradients of empirical loss and regularization term is obtuse ( \( \gt \!\!90^\circ \) ). As shown in Figure 2(a), the effect of regularization term forms a direction that might slow down the empirical loss descending. As shown in Figure 2(b), GrOD decomposes the gradient of regularization term and truncates the conflicted direction for the actual descent direction estimation. On the other hand, the angle between the actual descent direction and regularization gradient and the angle between the actual descent direction and empirical loss gradient are both acute ( \( \le \!\!90^\circ \) ), so as to secure the regularization effect while ensuring empirical loss descending. In this way, we can understand GrOD as the solution to the relaxed constraint programming problem for ERM-preserved descent direction estimation addressed in Section 3.2. Furthermore, in this example, due to the truncated direction of the regularizer’s gradient, the GrOD descent direction cannot be achieved by any linear combinations of ERM loss gradient and the regularizer’s gradient.

In transfer learning and knowledge distillation experiments, we used the ResNet-18 [56] as our base model with three widely used source datasets including ImageNet [8], Places 365 [57], and Stanford Dogs 120 [58] for weights pre-training. To evaluate the performance of transfer learning, we selected four datasets as the target datasets. These are Caltech 256 [5], MIT Indoors 67 [6], Flowers 102 [59], and CIFAR-10 [7]. Note that, we follow the same settings used in [2] for Caltech 256 setup, where 30 or 60 samples randomly drawn from each category for training with 20 remaining samples for testing. In adversarial training experiments, we used a small model consisting of two CNN layers and two fully connected layers with Fashion MNIST [60], while the ResNet-18 with CIFAR-10. Table 1 presents the statistics on some basic facts of all the datasets used in experiments.

In this section, we report the results of overall performance comparison based on the above tasks using \( L^2 \) -SP [2] and its variant based on GrOD for knowledge transfer from pre-trained models. We primarily focus on evaluating the performance improvement contributed by GrOD on top of \( L^2 \) -SP, comparing to the vanilla implementations. Both source and target tasks are trained on a typical ResNet-18 architecture. The knowledge transfer from ImageNet to all target tasks seems all good, as ImageNet contains more than 1,000 classes of images with more categories covered and rich features offered. However, the performance of knowledge transfer from Stanford Dog 120 to MIT Indoor 67 might be quite limited or even negatively affected the learning procedure, as these two datasets contain quite different images—dogs v.s. indoor scenes. Further discussion on the negative transfer effects would be addressed in Section 4.2.2.

We report the results of overall performance comparison based on the aforementioned tasks using Feature-wise Knowledge Distillation [3, 22] and its variant based on GrOD for Teacher–Student training of deep neural networks. We also focus on evaluating the performance improvement contributed by GrOD on top of Knowledge Distillation (denoted as “KnowDist” in this article), comparing to [3]. We use a ResNet-18 pre-trained on ImageNet as the Teacher Network.

We present the overall accuracy comparisons in Table 3. GrOD improves the performance of Teacher–Student training in all Student networks (also ResNet-18) training. For example, to train a Student network using CIFAR-10 dataset, common Knowledge distillation achieves 96.43% accuracy while GrOD further improves the accuracy to 96.57%. These two numbers are quite closed to the state-of-the-art performance of ResNet-18 on CIFAR-10 datasets (without using additional training or data augmentation methods) [62]. As it has been shown in 3, GrOD brings significant improvement in all tasks. We test the performance of GrOD based on other Teacher networks (based on different datasets). GrOD achieves performance improvement in all cases, on top of [3].

In this section, we report the results of performance comparison based on Fashion MNIST and CIFAR-10 under, adversarial learning settings, using advt [4] and its variant based on GrOD. We also focus on evaluating the performance improvement contributed by GrOD on top of advt. We use a simple two-layer CNN⁴ and a ResNet-18 for this experiment.

We report the results of the following two case studies that provide further evidences supporting that GrOD works in the way that we assumed.

All in all, the performance improvement made by GrOD on top of L \( ^2 \) -SP [2], KnowDist [3], and Adversarial Learning [4] is quite marginal. However, we hope to point out that the performance improvements consistently exist in all cases, especially relieving the “negative transfer” where the use of regularizer hurts the transfer learning. Please note that, in our study, we include the experiments based on inappropriate pairs of source and target datasets/pre-trained models for knowledge transfer and knowledge distillation (e.g., in Tables 2 and 3) while most of existing works uses ImageNet as the source datasets or pre-trained models only.

With inappropriate pairs of source and target datasets/pre-trained models, the regularized learning with L \( ^2 \) -SP or Distll might hurts the performance compared to directly fine-tuning from pre-trained weights. In all negative transfer cases, GrOD improves the performance of regularized knowledge transfer or knowledge distillation while always achieving performance better than vanilla fine-tuning. Furthermore, for the adversarial training with Advt regularization [4], GrOD achieves better tradeoff between accuracy and robustness, with Pareto dominance in these two factors, under varying strength of perturbation. Again our theoretical analysis in Section 3.4 clearly states how GrOD could ensure the effectiveness of ERM learning procedure while preserving the regularization effects (Proposition 1), which solicits the performance improvement. Note that our theoretical analysis relies on two assumptions made in Section 3.1, we conducted case studies with experiments to validate these two assumptions empirically.

Though GrOD enjoys higher accuracy on average, in some cases, it also incorporates higher variances. For example, in Table 2, with weights pre-trained by ImageNet, GrOD( \( L^2 \) -SP) achieves 90.86% with 0.31% STD for the target task based on Flower 102, while \( L^2 \) -SP achieves 88.96% with 0.21% STD in the same settings. It is obvious that GrOD incorporates with higher variances, however the lower bound of confidence intervals of GrOD is still higher than the upper bound of confidence interval of the original algorithm. Furthermore, we also tried to hack the weight decay using GrOD, the results showed that GrOD cannot improve weight decay. (Note that the weight decay, i.e., the L \( ^2 \) -regularization, has been frequently considered as a stabilizer [63, 64] of the training procedure in a regularization of Ridge-style.) We believe both of these observations are due to the use of orthogonal decomposition on stochastic gradient. In practical deep learning, stochastic gradients—the noisy evaluation of loss functions’ derivatives, have been used to accelerate the learning procedure with mini-batch sampling. However, the gradient noise [65] after orthogonal decomposition might perturbate the training procedure and leads to instability.

In our experiments, to enable fair comparisons, we use hyper-parameters, including learning rates and the weights of regularizers, according to the default settings released from the open-source implementation of the algorithms [2, 3, 4] (most of which were tuned best through cross validations in their research). In the same set of experiments, both GrOD and the original algorithms used the same set of hyper-parameters, especially the weights of regularizers for fair comparisons. Note that the performance of GrOD could be further improved through tuning the hyper-parameters (rather than the use of default ones).

Furthermore, our theoretical analysis also shows that the descent direction of GrOD is not achievable through tuning the weight of regularizers’ term (Proposition 2). That means, no matter how one sets the hyper-parameters for vanilla regularized deep learning, the algorithm based on the vanilla loss gradient of regularized deep learning could not behave as same as GrOD.

Note that GrOD strategy is derived from the common stochastic gradient estimation used in stochastic gradient based learning algorithms, such as SGD, Momentum, conditioned SGD, Adam, and so on. It can be consider as an alternative approach for descent direction estimation on top of vanilla stochastic gradient estimation, where you can still use natural gradient-alike method to condition the descent direction or adopt Momentum-alike acceleration methods to replace the weight updating mechanism. We are not intending to compare GrOD with any gradient-based learning algorithms, as the contributions are complementary. One can freely use GrOD to improve any gradient-based optimization algorithms with the descent direction corrected.

Furthermore, according to the ERM-Effective descent direction assumption, GrOD can further lower the empirical loss while preserving regularization effects simultaneously, as the finally descent direction will be close to the both empirical loss gradient and regularizer gradient. Our later on experiment based on adversarial learning will show that no matter how regularizer weights are fine-tuned, GrOD can still outperform the traditional regularized deep learning algorithms that linearly combine the gradients of the two terms as the descent direction. In future work, we intend to study the asymptotic properties and convergence performance of GrOD, using Neural Tangent Kernel as the proxy [66] to lower the complexity in non-convex optimization analysis.

Please be advised that the regularized deep learning algorithms for transfer learning ( \( L^2 \) -SP) [2], knowledge distillation [3], and adversarial training [4] are not the state-of-the-art algorithms in the fields. In future work, we are interested in incorporating with more advanced methods, such as DELTA and its variants [24, 67], BSS [68], Co-Tuning [69], learning without forgetting [21], and deep ensemble learning [70], where more advanced and complicated regularizers have been proposed incorporating constrained features, singular value decomposition, category relationship, and so on. In future work, we would study the use of GrOD and its variants to improve regularized learning, especially focusing on applications other than image classification, such as segmentation & parsing [10, 12], regularized graphical learning [15, 16, 71], and network interpretability [72, 73].