Robust and Resource-Efficient Data-Free Knowledge Distillation
by Generative Pseudo Replay
Kuluhan Binici1, 2 , Shivam Aggarwal2 , Nam Trung Pham1 , Karianto Leman1 , Tulika Mitra2
1
arXiv:2201.03019v1 [cs.LG] 9 Jan 2022
2
Institute for Infocomm Research, A*STAR, Singapore
School of Computing, National University of Singapore
Abstract
Data-Free Knowledge Distillation (KD) allows knowledge transfer from a trained neural network (teacher) to
a more compact one (student) in the absence of original training data. Existing works use a validation set
to monitor the accuracy of the student over real data
and report the highest performance throughout the entire process. However, validation data may not be available at distillation time either, making it infeasible to
record the student snapshot that achieved the peak accuracy. Therefore, a practical data-free KD method should
be robust and ideally provide monotonically increasing
student accuracy during distillation. This is challenging because the student experiences knowledge degradation due to the distribution shift of the synthetic data.
A straightforward approach to overcome this issue is to
store and rehearse the generated samples periodically,
which increases the memory footprint and creates privacy concerns. We propose to model the distribution of
the previously observed synthetic samples with a generative network. In particular, we design a Variational
Autoencoder (VAE) with a training objective that is customized to learn the synthetic data representations optimally. The student is rehearsed by the generative pseudo
replay technique, with samples produced by the VAE.
Hence knowledge degradation can be prevented without storing any samples. Experiments on image classification benchmarks show that our method optimizes
the expected value of the distilled model accuracy while
eliminating the large memory overhead incurred by the
sample-storing methods.
Introduction
Recently there is a surging interest to deploy neural networks on the edge devices. Most of these devices have strict
resource (e.g., memory or power) constraints that are incompatible with the high computational demand of the neural
networks. Knowledge Distillation (KD) (Hinton, Vinyals,
and Dean 2015) is a well-studied approach to deploy pretrained neural networks on resource-constrained devices by
making them more compact. KD needs access to the original dataset that was used to train the teacher model. This is
challenging if the distillation is performed by a party other
than the model developers as they may not have access to the
Copyright © 2021, Association for the Advancement of Artificial
Intelligence (www.aaai.org). All rights reserved.
Figure 1: Example of student accuracy degradation over distillation steps due to catastrophic forgetting. Red vertical
line marks the epoch with peak accuracy on the validation
dataset.
original training dataset, either due to privacy issues or the
extremely large dataset size making its relocation infeasible.
Addressing this issue, some works utilize alternative data
that are publicly available (Addepalli et al. 2020; Nayak,
Mopuri, and Chakraborty 2021), while others propose to use
synthetic data. (Zhang et al. 2021; Ma et al. 2020) The second approach is called Data-Free KD and completely eliminates the need for any real data. The decoupling of the KD
from the original dataset allows distillation to be performed
in a much wider spectrum of scenarios in comparison to
the data-dependent approaches. Typically data-free KD approaches contain multiple rounds (epochs) of synthetic data
samples generation and knowledge transfer. In each round,
the synthetic samples are generated to close the current information gap between the teacher and the student. As the
student is trained and the gap gets smaller, the synthetic
sample distribution changes as well. Therefore, if the previously generated synthetic samples are not periodically rehearsed to the student, the information acquired in earlier
distillation epochs might be lost, causing accuracy degradation over time. This is known as catastrophic forgetting
(French 1999) Such accuracy degradation is not desirable
in practice, especially if the validation dataset is also not
available. The absence of the validation dataset prevents the
user from monitoring the student accuracy over time and
choosing the distilled model with the peak accuracy. For
example, Figure 1 shows the CIFAR10 validation accuracy
of the student model distilled by an existing work, DFAD
(Fang et al. 2019), over 100 epochs. The peak accuracy is
achieved at the 8th epoch and the accuracy degrades subsequently. As the users do not have access to the validation
dataset, they cannot evaluate the student accuracy and find
the peak. Hence, the final accuracy will be a function of the
arbitrary termination epoch chosen by the user. This suggests that an ideal data-free KD method should be robust
and either sustain high accuracy or monotonically increase
it over time so that the distillation can be safely terminated
after a pre-determined sufficient number of epochs. Recent
research (Yin et al. 2020; Fang et al. 2021) propose to store
the generated samples over epochs and continuously expose
the student to the entire collection at each distillation step.
However, with this approach, the memory overhead and the
wall-clock time per distillation step increase substantially.
If the dataset is complex in terms of the number of classes
and the samples it contains, the required number of distillation steps to achieve an accurate student model can be quite
large. For instance, for the Imagenet dataset, the DeepInversion method (Yin et al. 2020) stores 140K synthetic samples
that occupy around 17 GB and takes 2.8K NVIDIA V100
GPU-hours to complete distillation. Moreover, storing the
generated samples may violate the privacy considerations of
the original dataset as they can leak information related to
the original samples (Li and Zhang 2020). In summary, the
current data-free KD methods are either not robust in terms
of student accuracy throughout the distillation process or are
resource inefficient by storing the generated samples and can
have potential privacy concerns. This trade-off casts them
less appealing to the end-users.
In this work, we aim to democratize data-free KD by
proposing a novel method that preserves student accuracy
by performing replay without storing any data samples. We
achieve this by designing a Variational Auto-Encoder (VAE)
(Kingma and Welling 2013) to model the cumulative distribution of all the generated samples. We establish that the
vanilla VAE optimization objective is not suitable to model
the synthetic data distribution and propose a synthetic dataaware reconstruction loss. The modeled distribution is used
to infer “memory samples” that represent those generated
in earlier steps and rehearse the student with these samples
in conjunction with the newly generated samples to further
bridge the information gap between the teacher and the student. Hence, our approach has a constant and very limited
memory overhead of a few megabytes to store only the VAE
parameters and offers accurate distillation. Experimental results show that our method achieves up to 26.8% increase
in average student accuracy compared to methods without
replay on image classification benchmarks. Moreover, compared to methods that replay stored samples, ours can reduce
the memory footprint by gigabytes. Our concrete contributions can be summarized as follows.
• A novel data-free KD framework containing two specialized generators to allow the student network acquire new
knowledge while retaining the prior information.
• Enabling VAEs to operate with synthetic data, produced
for knowledge distillation, by modifying the reconstruction loss.
• Extensive experimental evaluation of our approach in
comparison with the state-of-the-art.
Related Work
Knowledge Distillation (KD)
KD (Hinton, Vinyals, and Dean 2015) can be defined as
transferring the learnt prediction behavior of a complex neural network model to a relatively smaller one. In literature,
the complex network is often referred as the “teacher” and
the compact network is referred as the “student”. Throughout the distillation process, the student is trained with the
guidance of the ground truth labels as well as the teacher’s
responses to the training data. The inclusion of teacher’s responses in the training objective consolidates the information provided to the student about the data-label relationships. Thus the student network can achieve much higher
accuracy than when it is trained with the supervision of only
the ground truth labels. This is viewed as a form of compression, as the compact student model approximates the teacher
by carrying more information than it could have learnt from
data on its own.
Data-Free KD
The concept of performing knowledge distillation using synthetic data samples in place of a real dataset is called datafree KD. Unavailability of real data with ground truth labels
limits the student training to be guided only by the teacher
outputs (softmax logits, activation maps, etc.). One quality
that the synthetic samples are expected to have is that their
distribution should match that of the original data (Yoo et al.
2019; Nayak et al. 2019; Chen et al. 2019; Haroush et al.
2020). Another one is that they should be optimal to close
the information gap between the teacher and the student
(Micaelli and Storkey 2019; Fang et al. 2019). Producing
samples that satisfy both, generally yields the best distillation results (Yin et al. 2020; Fang et al. 2021; Binici et al.
2021). During distillation, the synthetic data distribution is
updated at each epoch to achieve the above-mentioned qualities. Such distribution shift could cause the student model
to lose the information over epochs and experience accuracy degradation (Binici et al. 2021). This could be avoided
by periodically rehearsing the student with samples from all
the distributions it has previously observed. We can group
the existing data-free KD methods based on whether they
employ such practice or not as replay-based methods and
replay-free methods.
The replay-based methods typically store a collection of
generated synthetic samples over epochs. CMI (Fang et al.
2021) is a recent work that utilizes the model inversion technique to generate the synthetic samples and store them all
in memory. Each time a newly generated batch is added,
the student is trained with the entire collection of samples.
This approach suffers from high memory utilization and requires a significant amount of time to achieve high-quality
distillation. (Binici et al. 2021) proposed to use a fixed-sized
b)
a)
Sample Generation
n
loss
m
db
ate
loss
Novel sample
batch
z∼
h
atc
Knowledge Distillation
tan
ce
T(x)l−1
n
co
T(x)l
Ge
ne
(0,1)
yT̂
Generator (novel)
rat
or
ba
(no
ck
ve
n(z; θ n)
l)
pr
op
ag
atio
n
KD loss
r
Teacher T(x; θT )
to
era
en
lg
ve loss
tch
le
En
co
mp
no
sa
l
ve
de
ba
no
r
−K
ba
ck
Encoder
Generator (memory)
m(z; θ
ℰm(x; θℰm)
Ge
yŜ
)
ne
VA
E
m
z∼
(T (
op
ag
rat
atio
or
x) ,
l T
n
n
em
ory
tch
)
mp
tio
uc
str
s
los
rec
r
mo
yT̂ |
...
| ŷ
S)
)
on
ba
a
ys
eD
ion
t
illa
ist
me
Student S(x; θS)
L(
yT̂
(x)
l−
1
(m
le
Memory sample
batch
pr
dg
wle
o
Kn
(0,1)
Figure 2: Method overview. (a) Proposed PRE-DFKD framework. The student and the generators are trained alternately. First
the generators are fixed and the student is trained by a combination of novel and memory samples. In the next stage, student
is fixed and the generators are trained by the learning signals received from the KD. (b) Training process of VAE. First, the
memory generator is frozen and a batch of memory samples are inferred. Later the memory batch is combined with a novel
sample batch to train the encoder-decoder pair.
memory buffer instead of storing all synthetic samples. Although the memory overhead is constant, this approach does
not completely eliminate the possibility of catastrophic forgetting as the student is rehearsed only with a subset of previously observed samples.
The replay-free approaches train the student only with the
newly generated samples at any distillation epoch. DAFL
(Chen et al. 2019) introduced three novel loss terms that
motivate the generated samples to be categorically diverse
and be classified with high confidence by the teacher.
DFAD (Fang et al. 2019) targeted generating samples that
would cause maximum information gain to the student when
learned. Since these works do not contain any replay mechanism, the student accuracy often fluctuates and degrades
over epochs.
Replay in Continual Learning
Continual Learning (CL) methods focus on incrementally
learning from a non-stationary dataset, with the goal of
mitigating catastrophic forgetting (Goodfellow et al. 2015),
i.e., the inability of the model to preserve the past knowledge while learning from the recent data. The CL methods (Mai et al. 2021) can be broadly classified into three
major approaches: regularization-based, architecture-based,
and replay-based (either generative or from a stored buffer),
with the latter being the most effective and the focus of this
paper. Vanilla replay-based techniques (Rebuffi et al. 2017)
maintain a buffer of raw samples from the past tasks and
replay them periodically with the new data in the learning
process. However, these methods become infeasible when
raw samples cannot be stored due to privacy or memory
constraints. Moreover, they cannot perform lifelong learning due to the inability to scale well with the growing data
stream. The generative replay strategies (Shin et al. 2017;
Wu et al. 2018) mitigate these shortcomings by training a
deep generative model to create pseudo-samples that mimic
the data distribution from past experiences. Our work leverages generative pseudo replay. In particular, we adopt a
VAE-based strategy (Shin et al. 2017) to preserve the knowledge of the previously generated synthetic samples. Unlike
existing continual learning works, our method focuses on
the generation of synthetic samples in a data-free environment. We propose a modified optimization objective to preserve the categorical information in reconstructed samples
and mitigate catastrophic forgetting in a data-free knowledge distillation scenario.
Proposed PRE-DFKD Approach
Our data-free KD approach is called Pseudo Replay Enhanced Data-Free Knowledge Distillation (PRE-DFKD).
Figure 2(a) provides an overview of PRE-DFKD. It consists
of data generation and knowledge distillation. In data generation, randomly sampled latent variables are transformed
by generative models to produce synthetic samples. Later
during knowledge distillation, the student model is trained
to categorize these samples similar to the teacher model.
These two stages repeat alternately until the target number
of steps is reached. We use two generative models in our
framework. The first one (novel sample generator) produces
samples that bring novel information to the student. The second generator (memory sample generator) is responsible for
re-exposing the student to the information acquired earlier.
Novel Sample Generation
Algorithm 1 Memory Sample Generator Training
We define novel samples as those that the student classifies differently from the teacher. When the student is trained
with these novel samples, it better approximates the teacher.
We condition our novel sample generator by including the
distance between student and teacher predictions in the optimization objective. To quantify such distance, we used
Jensen-Shannon (JS) divergence (see Eq. 3) (Yin et al. 2020)
as it is shown to provide better performing student models
compared to other alternatives (e.g. Kullback–Leibler, L1
norm) (Binici et al. 2021). Moreover, to ensure that the novel
sample distribution is similar to the original one, we also include the loss terms introduced by (Chen et al. 2019) (see
Eq. 2). The first two of these are predictive entropy and activation loss terms, which are minimized when the synthetic
data induce the teacher to output high valued activation maps
and low entropy prediction vectors. The third term, categorical entropy loss, sustains class balance in the generated
batches by maximizing the categorical entropy of synthetic
sample distribution. Eq. 1 shows the complete optimization
objective of the novel sample generator.
(T )
θn∗ := arg min Lφ + αLδ
(1)
θ
(T )
Lφ
=
1X
λ1 tiT log(ŷTi ) − λ2 fTi
n i
1
− λ3 H(p(ŷT ))
(2)
Lδ = 1 − JS(ŷT k ŷS ); z ∼ N (0, 1)
(3)
where θn∗ , represents the optimal values of the generator parameters denoted by θn . Moreover, ŷT = T (θ(z)) and fT
are the softmax outputs and the activation maps at the last
FC layer of the teacher T for the generated batch θ(z) respectively, while tiT = argmax(ŷTi ). H(p(ŷT )) denotes
the entropy of class label distribution in generated batch.
λi (i = 1, 2, 3) and α coefficients adjust the weighted contribution of each loss term.
Memory Sample Generation
As mentioned earlier, the distribution of novel samples shifts
over distillation steps. To prevent such shift from causing
the student to lose earlier learned information, we replay the
samples from earlier distributions. These memory samples
are inferred from a VAE that models the past synthetic data
distributions. Our memory sample generator is the decoder
part of the VAE and is trained jointly with the encoder. To
train the pair, we use a combination of novel samples and
memory samples. The inclusion of memory samples in the
training process is to prevent the generator itself from experiencing catastrophic forgetting. The process is described in
Algorithm 1 and visualized in Figure 2(b).
Incompatibility of Vanilla VAE loss for Synthetic Data
Representation Learning After conducting experiments
with vanilla VAE for pseudo replay in data-free KD, we observed that the teacher’s predictions of the memory samples
were of low confidence (14%). However, the average prediction confidence was significantly higher for novel samples (72%). This indicated that the modelled distribution
was not accurate. We posited that the training loss of the
INPUT: Novel sample generator Gn (z; θGn ), memory
sample generator Gm (z; θGm ), encoder Em (x; θEm ), batch
size b, latent vector dimension n.
# Train with novel samples
sample B vectors (z ∼ N (0, 1))
xn ← Gn (z)
ẑµ , ẑσ ← Em (xn )
zn ∼ N (ẑµ , ẑσ )
x̂n ← Gm (zn )
# Rehearse by reconstructing old samples
sample B vectors (zm ∼ N (0, 1))
xm ← Gm (zm )
z̄µ , z̄σ ← Em (xm )
z̄m ∼ N (z̄µ , z̄σ )
x̂m ← Gm (z̄m )
# Calculate & backpropagate loss
(i)
(i)
(i)
(i)
LV AE ← Lrec (xm , x̂m , xn , x̂n ) + γLKLD (z̄µ , z̄σ )
θGm , θEm ← optimizer.step(backward(LV AE ), θGm , θEm )
vanilla VAE is incompatible with the properties of synthetic samples. The reconstruction term in the loss function
2
(Lrec = kx − D(E(x))k2 ) is optimized when the decoded
samples D(E(x)) match the inputs x in pixel space. If the
L2 distance between the input and target image pixels is low,
the loss term yields a small value. This assumes that small
differences in pixel values do not impact the information
content significantly. While this assumption holds for real
images, it might not be accurate for synthetic ones. The synthetic samples are generated for the sole purpose of knowledge transfer. They are not visually realistic (see Figure 3)
yet still achieve their goal. Therefore, the assumption that
two images that are visually similar represent the same content may not hold for synthetic samples. To test this hypothesis, we inject random Gaussian noise to both real and synthetic samples and record the percentage of samples that preserved their class labels. The experimental details are given
in the appendix. The effect of pixel-wise perturbations on
class labels is visualized in Figure 3. It can be seen that the
class information of synthetic samples got affected more significantly than real samples. This suggests that even if a reconstructed sample yields a small reconstruction loss with
very similar pixel values to those of the synthetic input, its
categorical information could be completely different.
Synthetic Data-Aware VAE Reconstruction Loss To ensure that the categorical information is preserved, we modify
the VAE reconstruction loss. Our synthetic data-aware loss
constrains the reconstructed samples to cause similar feature
responses in the last layers of the teacher. Typically, these
layers are considered to represent content-related information in Convolutional Neural Networks (CNNs) (Jing et al.
2019). To impose the mentioned constraint, we add the term
in Eq. 4 to the VAE optimization objective.
X
l∈L
|T (x)l − T (D(E(x)))l |1
(4)
Algorithm 2 Memory Sample Generator Inference
INPUT: memory sample generator Gm (z; θGm ), teacher
T (x; θT ).
OUTPUT: memory batch xm .
# Tune latent variables
sample B vectors (zm ∼ N (0, 1))
xm ← Gm (zm )
ŷT ← TP
(xm )
R ← n1 i tiT log(ŷTi ) + KL(N (µz , σz ) k N (0, 1))
Lz ← −H(p(ŷT )) + R
zm ← optimizer.step(backward(Lz ), z)
# Infer memory samples
xm ← Gm (zm )
Figure 3: Example real and synthetic CIFAR10 samples are
given before and after noise injection in the first row. In the
second row, the projection of real and synthetic samples injected with noise on 2-dimensional planes are shown. Colors
represent the class labels assigned to these samples by the
teacher model. The original samples are marked with purple
crosses.
The encoder and decoder networks are represented by D and
E respectively. Moreover, the set L denotes the selected set
of teacher network (T ) layers. The total training loss we use
can be denoted as,
Lrec
LV AE = Lrec + LKLD
(5)
X
= |x−D(E(x))|1 +
|T (x)l −T (D(E(x)))l |1 (6)
l∈L
LKLD = (N (µz , σz ) k N (0, 1))
(7)
where x is the input sample batch; µz and σz are the mean
and variance of the latent vector (z = E(x)) distribution.
Sustaining Class Balance in Inferred Memory Batches
Once we accomplish training a VAE with synthetic data, another remaining issue is to ensure that the inferred memory
samples are evenly distributed into classes. Typically, latent
vectors that are sampled from standard Gaussian distribution
are fed to the generator to produce new images. However,
our task requires the generated batch to be diverse in terms
of the categorical contents of the samples. This is to guarantee that the student learns to approximate the teacher for
samples from any category. Therefore, to provide such class
balance, we freeze the memory sample generator and tune
the input latent vectors.
∗
zm
:= arg
max
z ∼N (0,1)
(p(ŷN )log(p(ŷN )))
(8)
The procedure we used to solve the optimization objective
described in Eq. 8 is given in algorithm 2. Tuning the input
latent variables can deviate their distribution from standard
normal, which would cause the generator to produce noise.
To prevent such deviation, we use the Kullback–Leibler
(KL) divergence as a regularization term. Moreover, to sample the most representative images (within each class) from
the modeled distribution, we also include the predictive entropy loss in the regularization term (R).
Knowledge Distillation
Knowledge transfer happens by minimizing the distance
between teacher and student predictions against combined
batches of novel and memory samples. The minimization
problem that results in optimal student model parameters is
θS∗ := arg min kS((xn , xm ); θS ) − T ((xn , xm ); θT )k1
θS
(9)
θS∗ in Eq. 9 denotes the optimal student model parameters;
xn and xm are novel and memory samples, respectively.
Experimental Evaluation
We demonstrate the effectiveness of PRE-DFKD in improving the expected student accuracy and reducing resource utilization on several image classification benchmarks. We provide a comparison with methods that store samples for replay and those that do not employ replay. Our baselines from
the first category are CMI (Fang et al. 2021), and the memory bank approach (Binici et al. 2021) that we refer to as
MB-DFKD. For the baselines that do not use replay, we select DAFL (Chen et al. 2019) and DFAD (Fang et al. 2019)
methods. For a fair comparison, we used the implementations of CMI, DAFL, and DFAD available from the authors’
GitHub pages.
Datasets: We use four datasets with different complexities and sizes. The simplest is MNIST (LeCun et al. 1998)
that contains 32 × 32 grayscale images from ten classes. CIFAR10 (Krizhevsky, Hinton et al. 2009) contains RGB images from ten classes (3 × 32 × 32). CIFAR100 (Krizhevsky,
Hinton et al. 2009) contains hundred different categories
while the samples have the same dimensions as those in CIFAR10. Lastly, the most complex dataset we use is Tiny ImageNet (Deng et al. 2009) that contains 64 × 64 RGB samples from 200 classes.
Method
Train with data
DAFL
DFAD
CMI
MB-DFKD
PRE-DFKD (ours)
MNIST
T :LeNet5 (98.9%)
S :LeNet-half
µ
σ 2 accmax
98.7 0.5 98.9
87.3 6.6 98.2
63.5 6.8 98.3
memory: N.A.
memory: 6.7 MB
88.6 3.2 98.3
memory: 2.1 MB
90.3 1.9 98.3
CIFAR10
T :ResNet34 (95.4%)
S :ResNet18
µ
σ2
accmax
89.0 8.1
95.2
62.6 17.1 92.0
86.1 12.3 93.3
memory: 250 MB
82.4 16.6 94.8
memory: 20 MB
83.3 16.4 92.4
memory: 2.1 MB
87.4 10.3 94.1
CIFAR100
T :ResNet34 (77.9%)
S :ResNet18
µ
σ2
accmax
71.3 8.1
77.1
52.5 12.8 74.5
54.9 12.9 67.7
memory: 500MB
55.2 24.1 77.0
memory: 20 MB
64.4 18.3 75.4
memory: 2.1 MB
70.2 11.1 77.1
Tiny ImageNet
T :ResNet34 (71.2%)
S :ResNet18
µ
σ2
accmax
60.2 8.8
64.9
39.5 10.3 52.2
memory: 2.7 GB
memory: 20MB
45.7 11.5 53.5
memory: 2.1 MB
46.3 11.0 54.2
Table 1: Student accuracy results for Data-Free KD on four image classification benchmarks. The T and S values denote the
teacher-student pairs. The values for µ and σ 2 represent mean and variance (%) of the averaged student validation accuracy
(over 4 runs), over epochs. accmax is the maximum recorded accuracy at any epoch of any distillation run.
Implementation Details We run each method for 200,
200, 400, and 500 epochs for MNIST, CIFAR10, CIFAR100,
and Tiny ImageNet, respectively. To evaluate each dataset
and method pair, we conduct four runs. For MNIST, we
select LeNet5 (LeCun et al. 1998) and LeNet5-half as the
teacher-student pair. For the remaining datasets, we use
ResNet34 (He et al. 2016) as the teacher and ResNet18 as
the student. While benchmarking the baseline methods, we
use the hyper-parameter configurations from the papers or
GitHub pages where available. Additional experimental details are given in the appendix. We note that CMI failed
for MNIST and Tiny ImageNet datasets. The original implementation required the teacher models to contain Batch
Normalization layers and as LeNet does not contain any, we
could not test it on MNIST. For Tiny ImageNet, although we
searched extensively for proper hyper-parameter configuration, none of our trials yielded comparable results. Similarly,
all our experiments with DFAD on Tiny ImageNet were unsuccessful.
Evaluation Metrics: If we consider the termination step
of the distillation as a random variable ts ∈ R, then our goal
is
max E[accts ] − σ 2 [accts ]
(10)
ts
where accts stands for student accuracy at epoch ts. To report results, first, we average the accts across all runs at the
corresponding epoch (ts). These averaged accuracy series
(i)
(i)
can be denoted as Ei [accts ] where accts is the student accuracy observed in the ith run at epoch ts. We report the
(i)
mean and variance values of Ei [accts ]. The mean (µ) corre(i)
sponds to Ets [Ei [accts ]] and the variance (σ 2 ) corresponds
(i)
to σ 2 [Ei [accts ]]. Additionally, we include the peak accuracy recorded across all runs (accmax ) to demonstrate that
our method achieves state-of-the-art performance when the
validation set is available.
Improved Distillation Quality: Table 1 provides a summary of the results across four runs. The accuracies of the
pre-trained teachers are given next to T in Table 1 followed by the student accuracy with different methods. The
first row Train with data shows the student accuracy when
trained with the original dataset. The remaining rows report the accuracy for different baseline data-free KD methods and our approach (PRE-DFKD). As expected, the Train
with data student accuracy is lower than the teacher accuracy and the data-free methods have lower student accuracy
than Train with data student accuracy. PRE-DFKD achieves
student accuracy that is better than almost all the baseline
data-free KD approaches and approximates the student accuracy trained with the original dataset closer for all datasets.
When compared with baselines that do not contain replay
(DAFL and DFAD), PRE-DFKD is more robust for validation set-agnostic distillation with higher expectation and
lower variance. When compared with CMI and MB-DFKD
that store generated samples for replay, PRE-DFKD typically is at least similar and more often better. It might be
counter-intuitive that although CMI stores and replays all
generated samples, PRE-DFKD outperforms it. This is because the accuracy gain rate of the student models distilled
via CMI is relatively low (see Figure 4) causing lower expected value. We believe the gain rate is low because the
novel samples constitute only a small proportion when compared to the replay samples. Lastly, MB-DFKD performing lower than PRE-DFKD suggests that limited amount of
stored samples in MB-DFKD is not as effective as modeling
the entire distribution of earlier observed samples in PREDFKD.
Furthermore, to visualize the effect of pseudo replay on
student accuracy throughout distillation, we plot the student accuracy curves (see Figure 4) for the MNIST and CIFAR100 benchmarks. For all methods, the observed accu(i)
racy series (accts ) across different runs on MNIST benchmark, had relatively higher variation than on other benchmarks. Therefore, for MNIST, we plot the accuracy curves
with mean and variances across runs are indicated. For
CIFAR100, we only display the mean accuracy curve (µ)
across four runs for each baseline.
Figure 4: Visualized student accuracy curves. The first row contains MNIST results with mean and variance values over runs.
The second row contains student accuracies averaged over runs for CIFAR100.
Reduced Memory Footprint: We next compare PREDFKD with baselines using replay in terms of memory footprint. The additional memory required for the replay are
noted in Table 1. Although CMI failed in Tiny ImageNet experiments, we report the memory it utilized after running for
the same number of epochs (500) as other methods. The Table 1 shows that PRE-DFKD significantly reduces the memory footprint for replay from hundreds of megabytes and
even gigabytes to a constant 2MB, that is the memory occupied by the VAE parameters. Such reduction becomes more
significant compared to CMI as the number of distillation
steps increases since more batches of samples are generated
and stored. The calculated memory footprint of CMI for different choices of distillation steps ts and synthetic data dimensions are marked in Table 2. As the VAE architecture is
independent of the selection of teacher-student pairs and distillation steps, the memory overhead does not scale based on
them. Although MB-DFKD maintains fixed-sized storage,
the required storage for optimal distillation largely depends
on the dataset. If the storage is small relative to the number
of distillation steps, the effectiveness of replay diminishes.
pixel dim. \ts
32
64
128
2000
2.5 GB
10 GB
40 GB
4000
5 GB
20 GB
80 GB
8000
10 GB
40 GB
160 GB
16000
20 GB
80 GB
320 GB
Table 2: CMI memory footprint for various pixel dimensions
and distillation epochs
Ablation Study: We now validate that the pseudo replay
is effective regardless of the novel sample synthesis strategy.
To do this, we couple our memory generator to the DFAD
baseline method. In Figure 5, DFAD distilled student accuracy is plotted with and without the inclusion of our memory generator. We name the VAE-enhanced baseline PRE-
DFAD. For a fair comparison, we fix the random seeds for
all the runs. The plots indicate that pseudo replay succeeded
in preventing accuracy degradation caused by catastrophic
forgetting even for DFAD. Additionally, we plot the PREDFKD results where the proposed class-balanced memory
sample inference and synthetic data-aware reconstruction
loss are omitted. This way, we examine the individual contributions of these techniques. In Figure 5, it is apparent that
the exclusion of any of these impaired the student accuracy
progressions.
Figure 5: Ablation experiment results on MNIST.
Conclusion
In this work, we propose a data-free KD method that prevents the distilled model accuracy from degrading over time
by generative pseudo replay. While our method is more robust than existing approaches in the absence of validation
data, it also significantly reduces the memory footprint and
improves privacy preservation. We believe these contributions provide a step towards allowing a wider audience to
benefit from data-free KD, i.e., democratize it. In the future,
our approach can be further improved by exploring ways to
reduce the impact of dataset-specific hyper-parameter selection on distillation performance.
Acknowledgement
This work was supported by the A*STAR Computational
Resource Centre through the use of its high-performance
computing facilities and by the National Research Foundation, Singapore under its Competitive Research Programme
Award NRF-CRP23-2019-0003.
Li, Z.; and Zhang, Y. 2020. Membership Leakage in LabelOnly Exposures. arXiv preprint arXiv:2007.15528 .
Ma, X.; Shen, Y.; Fang, G.; Chen, C.; Jia, C.; and Lu, W.
2020. Adversarial Self-Supervised Data-Free Distillation
for Text Classification. arXiv preprint arXiv:2010.04883 .
References
Addepalli, S.; Nayak, G. K.; Chakraborty, A.; and Radhakrishnan, V. B. 2020. DeGAN: Data-Enriching gan for retrieving representative samples from a trained classifier. In Proceedings of the AAAI Conference on Artificial Intelligence,
volume 34, 3130–3137.
Binici, K.; Pham, N. T.; Mitra, T.; and Leman, K. 2021. Preventing Catastrophic Forgetting and Distribution Mismatch
in Knowledge Distillation via Synthetic Data. arXiv preprint
arXiv:2108.05698 .
Chen, H.; Wang, Y.; Xu, C.; Yang, Z.; Liu, C.; Shi, B.; Xu,
C.; Xu, C.; and Tian, Q. 2019. Data-free learning of student
networks. In ICCV, 3514–3522.
Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and FeiFei, L. 2009. Imagenet: A large-scale hierarchical image
database. In 2009 IEEE conference on computer vision and
pattern recognition, 248–255. Ieee.
Fang, G.; Song, J.; Shen, C.; Wang, X.; Chen, D.; and Song,
M. 2019. Data-Free Adversarial Distillation. arXiv preprint
arXiv:1912.11006 .
Fang, G.; Song, J.; Wang, X.; Shen, C.; Wang, X.; and Song,
M. 2021. Contrastive Model Inversion for Data-Free Knowledge Distillation. arXiv preprint arXiv:2105.08584 .
French, R. M. 1999. Catastrophic forgetting in connectionist
networks. Trends in cognitive sciences 3(4): 128–135.
Goodfellow, I. J.; Mirza, M.; Xiao, D.; Courville, A.; and
Bengio, Y. 2015. An Empirical Investigation of Catastrophic
Forgetting in Gradient-Based Neural Networks.
Haroush, M.; Hubara, I.; Hoffer, E.; and Soudry, D. 2020.
The knowledge within: Methods for data-free model compression. In CVPR, 8494–8502.
He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual
learning for image recognition. In CVPR, 770–778.
Hinton, G.; Vinyals, O.; and Dean, J. 2015. Distilling the knowledge in a neural network. arXiv preprint
arXiv:1503.02531 .
Jing, Y.; Yang, Y.; Feng, Z.; Ye, J.; Yu, Y.; and Song, M.
2019. Neural style transfer: A review. IEEE transactions on
visualization and computer graphics 26(11): 3365–3385.
Kingma, D. P.; and Welling, M. 2013. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 .
Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple
layers of features from tiny images .
LeCun, Y.; Bottou, L.; Bengio, Y.; and Haffner, P. 1998.
Gradient-based learning applied to document recognition.
Proceedings of the IEEE 86(11): 2278–2324.
Mai, Z.; Li, R.; Jeong, J.; Quispe, D.; Kim, H.; and Sanner,
S. 2021. Online Continual Learning in Image Classification:
An Empirical Survey.
Micaelli, P.; and Storkey, A. J. 2019. Zero-shot knowledge
transfer via adversarial belief matching. In NIPS, 9551–
9561.
Nayak, G. K.; Mopuri, K. R.; and Chakraborty, A. 2021. Effectiveness of arbitrary transfer sets for data-free knowledge
distillation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 1430–1438.
Nayak, G. K.; Mopuri, K. R.; Shaj, V.; Radhakrishnan, V. B.;
and Chakraborty, A. 2019. Zero-shot knowledge distillation
in deep networks. In International Conference on Machine
Learning, 4743–4751. PMLR.
Rebuffi, S.-A.; Kolesnikov, A.; Sperl, G.; and Lampert, C. H.
2017. iCaRL: Incremental Classifier and Representation
Learning. 2017 IEEE Conference on Computer Vision and
Pattern Recognition (CVPR) 5533–5542.
Shin, H.; Lee, J. K.; Kim, J.; and Kim, J. 2017. Continual
Learning with Deep Generative Replay.
Wu, Y.; Chen, Y.; Wang, L.; Ye, Y.; Liu, Z.; Guo, Y.; Zhang,
Z.; and Fu, Y. 2018. Incremental Classifier Learning with
Generative Adversarial Networks. CoRR abs/1802.00853.
URL http://arxiv.org/abs/1802.00853.
Yin, H.; Molchanov, P.; Alvarez, J. M.; Li, Z.; Mallya, A.;
Hoiem, D.; Jha, N. K.; and Kautz, J. 2020. Dreaming to
distill: Data-free knowledge transfer via DeepInversion. In
CVPR, 8715–8724.
Yoo, J.; Cho, M.; Kim, T.; and Kang, U. 2019. Knowledge
extraction with no observable data. In NIPS, 2705–2714.
Zhang, Y.; Chen, H.; Chen, X.; Deng, Y.; Xu, C.; and Wang,
Y. 2021. Data-Free Knowledge Distillation for Image SuperResolution. In Proceedings of the IEEE/CVF Conference on
Computer Vision and Pattern Recognition, 7852–7861.
Appendix
Additional Implementation Details
This section contains additional details related to the experimental setup. For all datasets,
• Both generators are updated with Adam optimizer.
• Input vectors of the memory generator are tuned with
learning rate same as lrGn .
• λ1 , λ2 , λ3 and α coefficients are set as 1, 5, 0.1 and 1
respectively
• PyTorch framework is used.
• Experiments are done on Linux version 4.18.0147.el8.x86 64
MNIST: In MNIST experiments, we set the learning rates
of the student (lrS ) and the novel sample generator (lrGn ) as
0.004 and 0.2. We use Adam optimizer to update the student.
The learning rate for the memory sample generator (lrGm )
1
times lrGn . We set the novel and memory sample batch
is 40
sizes as 512 and 256 respectively. We use 100-dimensional
latent vectors as inputs to the generators. The whole distillation process of 200 epochs took 1.3 hours to complete on 2
NVIDIA V100 GPUs.
CIFAR10: In CIFAR10 experiments, we set (lrS ) and
lrGn as 0.1 and 0.02 respectively. We use SGD optimizer to
1
update the student. lrGm is 10
times lrGn . We set the novel
and memory sample batch sizes as 1024 and 128 respectively. We used 1000-dimensional latent vectors as inputs to
the generators. The whole distillation process of 200 epochs
took 11.6 hours to complete on 2 NVIDIA V100 GPUs.
CIFAR100: In CIFAR100 experiments, we set (lrS ) and
lrGn as 0.1 and 0.02 respectively. We use SGD optimizer to
1
update the student. lrGm is 10
times lrGn . We set the novel
and memory sample batch sizes as 1024 and 128 respectively. We used 1000-dimensional latent vectors as inputs to
the generators. The whole distillation process of 400 epochs
took 24 hours to complete on 2 NVIDIA V100 GPUs.
Tiny ImageNet: In Tiny ImageNet experiments, we set
(lrS ) and lrGn as 0.1 and 0.02 respectively. We use SGD
1
times lrGn . We
optimizer to update the student. lrGm is 10
set the novel and memory sample batch sizes as 1500 and
187 respectively. We used 2000-dimensional latent vectors
as inputs to the generators. The whole distillation process of
500 epochs took 29 hours to complete on 2 NVIDIA V100
GPUs.
Synthetic Sample Class Label Sensitivity Against
Pixel-Space Perturbations
The reconstructed outputs of a VAE could be viewed as
Xs + ǫ for the source image Xs . To measure the degree that
the introduced error (ǫ) affects the content information of
synthetic samples, we simulate the reconstruction by injecting noise to synthetic novel samples and observe the class
labels that the teacher assigns to these noisy versions. Here
we assume that ǫ comes from Gaussian distribution and sample the injected noise from the Guassian distribution with 0
mean and 0.2 variance (N (0, 0.2)). We sample one image
from the real CIFAR10 training set and one synthetic image
from a generator that is optimized to replicate CIFAR10 images. Later we sample 1000 ǫ values from N (0, 0.2) and perturb both images with them. This can be viewed as producing samples from the neighborhoods of the original images
in pixel space. All 1000 of the noisy images obtained from
the original real image had the same class label. Whereas,
the original class label was preserved for only 165 images
out of 1000 obtained from the synthetic sample. This suggests that the content information of synthetic samples gets
significantly more affected by the reconstruction error than
the real samples.