Learning Accurate Performance Predictors for Ultrafast Automated Model Compression

Abstract

In this paper, we propose an ultrafast automated model compression framework called SeerNet for flexible network deployment. Conventional non-differen-tiable methods discretely search the desirable compression policy based on the accuracy from exhaustively trained lightweight models, and existing differentiable methods optimize an extremely large supernet to obtain the required compressed model for deployment. They both cause heavy computational cost due to the complex compression policy search and evaluation process. On the contrary, we obtain the optimal efficient networks by directly optimizing the compression policy with an accurate performance predictor, where the ultrafast automated model compression for various computational cost constraint is achieved without complex compression policy search and evaluation. Specifically, we first train the performance predictor based on the accuracy from uncertain compression policies actively selected by efficient evolutionary search, so that informative supervision is provided to learn the accurate performance predictor with acceptable cost. Then we leverage the gradient that maximizes the predicted performance under the barrier complexity constraint for ultrafast acquisition of the desirable compression policy, where adaptive update stepsizes with momentum are employed to enhance optimality of the acquired pruning and quantization strategy. Compared with the state-of-the-art automated model compression methods, experimental results on image classification and object detection show that our method achieves competitive accuracy-complexity trade-offs with significant reduction of the search cost. Code is available at https://github.com/ZiweiWangTHU/SeerNet.

Appendices

Appendix

Appendix A: Mathematical formulation from (11) to (12) of the manuscript.

Since we leverage the deterministic neural networks to predict the accuracy of various lightweight models, we present the alternative objective function to tractably calculate the objective (11) in the manuscript based on importance sampling. We first rewrite the importance weight in (11) of the manuscript via the analytical form of Dirac-delta function:

$$\begin{aligned} \qquad \qquad \qquad \frac{p(a|\varvec{s})}{p(a|\varvec{\hat{s}})}=\frac{\lim \limits _{\epsilon \rightarrow 0}\frac{1}{\pi }\frac{\epsilon }{\epsilon ^2+(a-a_{10})^2}}{\lim \limits _{\epsilon \rightarrow 0}\frac{1}{\pi }\frac{\epsilon }{\epsilon ^2+(a-a_{20})^2}} \end{aligned}$$

(16)

where $a_{10}$ and $a_{20}$ are distribution parameters of original and perturbed policies parameterized by the performance predictor. As $\epsilon$ is higher-order infinitesimal of $a-a_{10}$ and $a-a_{20}$ according to the definition of Dirac-delta function, the importance weight can be rewritten as follows:

$$\begin{aligned}&\frac{p(a|\varvec{s})}{p(a|\varvec{\hat{s}})}=\frac{(a-a_{20})^2}{(a-a_{10})^2}\nonumber \\&=\frac{(a-a_{10})^2+(a_{10}-a_{20})^2+2(a-a_{10})(a_{10}-a_{20})}{(a-a_{10})^2} \end{aligned}$$

Since a means the predicted accuracy parameterized by $a_{10}$, the difference between a and $a_{10}$ can be assumed as a small constant $\gamma _{10}$ in the deterministic settings. Therefore, the difference between a and $a_{10}$ is far less than that between $a_{10}$ and $a_{20}$. We obtain the following relationship between $\frac{p(a|\varvec{s})}{p(a|\varvec{\hat{s}})}$ and $a_{10}-a_{20}$:

$$\begin{aligned} \qquad ~~ \frac{p(a|\varvec{s})}{p(a|\varvec{\hat{s}})}=(a_{10}-a_{20})^2/\gamma _{10} \propto (a_{10}-a_{20})^2 \end{aligned}$$

(17)

$a_{10}$ and $a_{20}$ are represented by $f(\varvec{s})$ and $f(\hat{\varvec{s}})$, and the loss function $l(f(\varvec{s}),a)$ of accuracy prediction is assigned with the $L_2$ norm of the difference between the predicted and actual accuracy $(f(\varvec{s}^i)-a^i)^2$. Therefore, we optimize the alternative objective (12) in the manuscript to provide informative supervision for performance predictor learning.

Fig. 6

The actual and predicted accuracy for different compression policy across various datasets and architectures, where random sampling and our active sampling are both evaluated. The architectures for evaluation on CIFAR-10 include VGG-small and ResNet20, and those on ImageNet contain MobileNet-V2, ResNet18 and ResNet50. The horizontal axis represents the predicted accuracy and the vertical axis means the actual accuracy. The mean squared errors (MSE) are also demonstrated in the figure

Fig. 7

The visualization of the optimal compression policies obtained by our ultrafast SeerNet with given computational cost constraint on image classification. We utilized VGG-small and ResNet20 architectures on CIFAR-10 and MobileNet-V2, ResNet18 and ResNet50 networks on ImageNet

Appendix B: The Actual and Predicted Accuracies for Sampled Lightweight Networks

We employed the architectures of VGG-small (Zhang et al., 2018) and ResNet20 (He et al., 2016) for automated model compression on CIFAR-10, and compressed MobileNet-V2 (Sandler et al., 2018), ResNet18 and ResNet50 architectures on ImageNet. We trained the performance predictor by the accuracy of 800 randomly sampled and 800 actively sampled compressed models respectively, and regressed the accuracy for 50 randomly sampled lightweight models via the well-trained performance predictor. We show the actual and predicted accuracy of random sampling and our active sampling in Fig. 6, where the MSE between the actual and predicted accuracy is also demonstrated. The predicted accuracy is generally closed to the actual one across the datasets, which shows the effectiveness of the performance predictor for automated model compression. Meanwhile, our active sampling strategy chooses the uncertain compression policies that provide informative supervision for performance predictor learning, so that the predicted accuracy is more precise compared with the random sampling strategy. Since the performance variance for different quantization and pruning strategies on largescale datasets is larger, and our active sampling strategy offers more benefits for the performance predictor learning on ImageNet. Although deeper architectures with large search space such as MobileNet-V2 and ResNet50 obtain higher MSE for their performance predictors, the active sampling policy is still capable of providing informative supervision for accurate performance predictor learning as the MSE is less than $5\times 10^{-4}$.

Table 7 The accuracy(%) and BOPs(G) variance for Table 1 in the manuscript by running experiments for 5 times

Table 8 The accuracy(%) and BOPs(K) variance for Table 2 in the manuscript acquired by running experiments for 5 times

Appendix C: Visualization of the Optimal Compression Policy

We show the bitwidth of weights and activations and pruning ratio across different layers for image classification in Fig. 7, where the computational cost constraint is 2.4G and 0.2G BOPs for compressing VGG-small and ResNet20 on CIFAR-10 and is 8G, 33G and 62G BOPs for MobileNet-V2, ResNet18 and ResNet50 compression on ImageNet respectively.

Because sparse networks require precise weights and activations to maintain the representational capacity and increasing the bitwidth of layers with high pruning ratio only brings slight computational cost, the layers with high pruning ratio are usually assigned with large bitwidth in the optimal compression policy. The optimal compression policy searched via the state-of-the-art method DJPQ (Wang et al., 2020b) only prunes the bottom layers since they sequentially prune the networks from bottom layers to top layers, while our SeerNet simultaneously optimizes the pruning strategy for all layers and preserves the informative channels with redundant channel removal.

For VGG-small and ResNet20 architectures trained on CIFAR-10, the bitwidth varies slightly across different layers. Meanwhile, the pruning ratio is high and the bitwidth is low for all layers, which means significant over-parameterization for both network architectures on CIFAR-10. The large bitwidth and low pruning ratio in the optimal compression policy for MobileNet-V2 indicates that the compact architecture is hard to be further compressed without sizable accuracy drop. On the contrary, ResNet50 is compressed with extremely low bitwidth, which demonstrates the significant redundancy. Moreover, the bitwidth of activations is usually larger than that of weights, which depicts that the model accuracy is more sensitive to activation quantization than weight quantization.

Appendix D: Implementation Details in Section 4.2.2

To validate the effectiveness of the presented performance predictor, we utilized the reinforcement learning and evolutionary algorithms to search the optimal compression policy via our performance predictor. For reinforcement learning, we leveraged the deep deterministic policy gradient (DDPG) (Lillicrap et al., 2015). Compared with (Wang et al., 2019a), we added an extra state $p_k$ representing the pruning ratio for the $k_{th}$ convolutional layer in the state space and supplemented an extra action $a_k^p$ to sample the pruning ratio of compression strategy in the action space. The accuracy of the compressed model applied in the reward function was obtained via our performance predictor. Other implementation details were the same as those in Wang et al. (2019a). For evolutionary algorithms, we followed the same implementation details in Wang et al. (2020a) to search the optimal compression policy except that we deleted the network architecture components for each candidate. The accuracy of each candidate applied in the fitness function was acquired via our performance predictor. We imposed the resource constraint by limiting the BOPs of the compressed models during the search process.

Table 9 The accuracy(%), MACs(K), BOPs(G) variance for Table 3 in the manuscript acquired by running experiments for 5 times

Table 10 The top-1 accuracy(%), MACs(G), BOPs(G) variance for Table 4 in the manuscript acquired by running experiments for 5 times

Table 11 The mAP(%), MACs(G), BOPs(G) variance for Table 5 in the manuscript acquired by running experiments for 5 times

Table 12 The mAP(%), MACs(G), BOPs(G) variance for Table 6 in the manuscript acquired by running experiments for 5 times

Table 13 The MACs(G), BOPs(G), top-1 classification accuracy and search cost on ImageNet with pruning-only methods in MobileNet-V2

Appendix E: Performance Variance of SeerNet

In order to show the performance variance of our SeerNet, we run SeerNet for 5 times including compression policy search and backbone training for results in Tables 1, 2, 3, 4, 5, 6. We report the mean and standard deviation of accuracies and computational complexity due to the variation, while the search cost and training cost are almost the same for each time. We leveraged the same complexity budget as that in Tables  1, 2, 3, 4, 5, 6 of the manuscript, and Tables 7, 8, 9, 10, 11, 12 show the experimental results with mean and standard deviation.

Appendix F: Performance of Pruning-only and Quantization-only Strategies

In this section, we evaluate SeerNet in the experimental settings with pruning-only and quantization-only strategies, where the backbone networks are only compressed by pruning and quantization policies in the above settings respectively. We implemented SeerNet following the details introduced in Section 4.1 of the manuscript except for the modifications that the compression policy sampling for performance predictor training only contains pruning or quantization for the two settings. The compared methods include AMC (He et al., 2018b), NetAdapt (Yang et al., 2018), MetaPruning (Liu et al., 2019) for pruning-only methods and contain HAQ (Wang et al., 2019a) and HAWQ (Dong et al., 2019) for quantization-only methods. For pruning-only strategies, since only one shared experimental setting exists in AMC, NetAdapt and MetaPruning which employs MobileNet-V2 with 0.22G MACs for evaluation, we also assign the similar complexity constraint for fair comparison. For quantization-only strategies, we compare SeerNet with HAQ and HAWQ with MobileNet-V2, ResNet18 and ResNet50. Tables 13 and 14 illustrate the results for pruning-only and quantization-only strategies respectively, where our SeerNet still outperforms the baseline methods by a sizable margin with much less marginal search cost.

Table 14 The MACs(G), BOPs(G), top-1 classification accuracy and search cost on ImageNet with quantization-only methods in MobileNet-V2, ResNet18 and ResNet50

Fig. 8

Several examples w.r.t. the compression policy complexity during the optimization with different policy initializations, where no optimization paths stop early because of exceeding the computational cost budget. Different colors represent various initializations, where the BOPs constraint is 30G

Table 15 The BOPs(G), top-1 classification accuracy and search cost on ImageNet with random search baseline and our SeerNet in ResNet18

Appendix G: Visualization of Policy Optimization

The presented barrier complexity loss in the compression policy optimization is amplified significantly for model complexity approaching the cost budget, which strictly limits the acquired pruning and quantization strategies within the complexity constraint. The complexity of the pruning and quantization policy in the optimization path usually keeps a margin from the computational cost budget. In order to empirically demonstrate the effectiveness of the barrier complexity loss, we implemented compression policy search for 50 times with different initialization, and the compression policies with complexity higher than the budget were never observed during the compression policy optimization. Figure 8 shows several examples w.r.t. the compression policy complexity during the optimization with different policy initializations, where no optimization paths stop early because of exceeding the computational cost budget.

Appendix H: Comparison with The Random Selection Baseline Method

To show the effectiveness of our search method, we conducted experiments to compare our SeerNet with random selection baseline method (RS) with ResNet18 on ImageNet. The pipeline of RS is demonstrated as follows: (a) randomly sampling k compression policies that satisfy the BOPs constraint, (b) exhaustively evaluating the acquired lightweight architectures, (c) selecting the one with the highest accuracy. With the same complexity constraint in Table 4 of the manuscript, we randomly sample 5 compression strategies that satisfy the BOPs budget for each random selection. Table 15 demonstrates the results, where the BOPs of RS is far from the budget and underperforms our SeerNet by a large margin regarding the accuracy.