GANs Can Play Lottery Tickets Too
Keywords: lottery tickets, GAN compression, generative adversarial networks
Abstract: Deep generative adversarial networks (GANs) have gained growing popularity in numerous scenarios, while usually suffer from high parameter complexities for resource-constrained real-world applications. However, the compression of GANs has less been explored. A few works show that heuristically applying compression techniques normally leads to unsatisfactory results, due to the notorious training instability of GANs. In parallel, the lottery ticket hypothesis shows prevailing success on discriminative models, in locating sparse matching subnetworks capable of training in isolation to full model performance. In this work, we for the first time study the existence of such trainable matching subnetworks in deep GANs. For a range of GANs, we certainly find matching subnetworks at $67\%$-$74\%$ sparsity. We observe that with or without pruning discriminator has a minor effect on the existence and quality of matching subnetworks, while the initialization weights used in the discriminator plays a significant role. We then show the powerful transferability of these subnetworks to unseen tasks. Furthermore, extensive experimental results demonstrate that our found subnetworks substantially outperform previous state-of-the-art GAN compression approaches in both image generation (e.g. SNGAN) and image-to-image translation GANs (e.g. CycleGAN). Codes available at https://github.com/VITA-Group/GAN-LTH.
One-sentence Summary: Winning tickets exist in the deep generative adversarial networks, which substantially outperform previous state-of-the-art compressed GAN
Code Of Ethics: I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics
Supplementary Material: zip
Code: [ VITA-Group/GAN-LTH](https://github.com/VITA-Group/GAN-LTH)
Data: [CIFAR-10](https://paperswithcode.com/dataset/cifar-10), [STL-10](https://paperswithcode.com/dataset/stl-10)
Community Implementations: [ 2 code implementations](https://www.catalyzex.com/paper/arxiv:2106.00134/code)
14 Replies
Loading