Flooding Regularization for Stable Training of
Generative Adversarial Networks

There are three categories in stabilization methods for the GAN training: change of adversarial loss type, regularization, and change of architectures [36].

Change of adversarial loss type Goodfellow et al. [9] pointed out that the BCE loss can lead to a problem that the discriminator can correctly discriminate the generated data ($D(G(\bm{z}))\simeq 0$), resulting in saturated gradients. To address it, a non-saturating BCE loss is proposed by modifying $f_{G}$ shown in Table 1.

However, the instability still remains [1, 28]. Goodfellow et al. [9]have shown that the training with the BCE loss minimizes the Jensen-Shannon divergence between the real data distribution $\mathbb{P}_{r}$ and generated data distribution $\mathbb{P}_{g}$. Arjovsky et al. [2] showed that the minimization causes instability, and they proposed Wasserstein loss based on Earth Mover (EM) distance. Moreover, Lim et al. [21] proposed a hinge loss, and Mao et al. [22] proposed a least squares loss based on minimizing Peason $\chi^{2}$ distance. Table 1 shows the losses. These methods can be regarded as integral probability metric (IPM)-based regularization [36], where the generators and discriminators belong to a particular function class, such as models with Lipschitz continuity.

Table 2 shows $L_{D_{\mathrm{opt}},\mathrm{real}}$, $L_{D_{\mathrm{opt}},\mathrm{fake}}$, $L_{D_{\mathrm{opt}}}$, and $L_{G_{\mathrm{opt}}}$ for each adversarial loss.

Regularization There are various regularization techniques to stabilize the GAN training. One major approach is the addition of regularization terms to the adversarial loss. This directly prevents gradient explosion and vanishing [18, 26], model overfitting [3, 2], and mode collapse [4]. Gradient penalty [10] is an example of adding a regularization term, which improves the discriminator stability by adding a squared error between the gradient norm and $1$ so that the gradient norm approaches $1$. It contributes to Lipschitz continuity and training stabilization. These methods require tuning of the coefficients to balance the adversarial loss. Label smoothing [28] regularizes through the target labels. However, its stabilization effect on adversarial losses other than the BCE loss is unknown. Normalization is another common approach to regularization [37, 24]. Spectral normalization [24], a representative normalization technique for GANs, stabilizes training with Lipschitz continuity of the discriminator through the normalization of the weight matrices. Unlike the existing regularization techniques for GANs, our method directly regularizes the adversarial loss.

Change of architecture Changing the architecture is a commonly used approach to stabilization. For example, when generating images with a high resolution, a method that efficiently preserves the entire image features is essential. Deep convolutional GAN (DCGAN) [27], which employs a convolutional layer, and self-atteintion GAN (SAGAN) [39], which introduces an attention mechanism, have been proposed. Some studies [15, 3, 16, 17] have proposed to generate high-resolution and photo-realistic images by devising architectures. These model changes are relatively easy to combine with loss changes because they do not disrupt the competing structure, which is represented by adversarial losses.

Adversarial architectures for an efficient high-dimension distribution generator training have been applied in many research fields. For instance, Mirza et al. [23] proposed conditional GANs to control generated images with labels. The idea was employed for a wide range of applications, such as text-to-image generation [40] and image-to-image translation [14]. Additionally, domain-adversarial neural network (DANN) [8], adversarial discriminative domain adaptation (ADDA) [35], and Wasserstein distance guided representation learning (WDGRL) [29] proposed the use of adversarial frameworks for domain adaptation, which alleviates the gap between source and target domain in classification tasks. These architectures also have challenges with training multiple models.

In supervised learning, overfitting is a well-known phenomenon in which a model performs well on training data but fails on unknown data. Typical methods for preventing overfitting in supervised learning are dropout [32], batch normalization [12], data augmentation [30], and label smoothing [33]. Many of these techniques can be applied not only to supervised learning but also to other domains, and they are sometimes adopted in GANs as well [20]. Ishida et al. [13] proposed flooding, a method that recalculates the loss based on the formula (4) with respect to the flood level $b$ so that the loss does not become extremely small and avoids overfitting. On the other hand, Xie et al. [38] proposed individual flood (iFlood) to apply flooding before taking the expected value of losses. The method is effective because it can regularize only the loss of overfitted instances. While these methods perform well in supervised learning, there is no standard for setting $b$, and it requires a tuning process.

GAN training has a generator $G$ and a discriminator $D$ and proceeds based on losses defined in Eq. (2). This section assumes simple GANs without any regularization term to simplify the discussion. For example, with the BCE loss [9], $L_{D}$ and $L_{G}$ can be written as

Instability in GANs can occur when the discriminator’s loss is too low, indicating overfitting of the discriminator. We aim to improve the stability by applying flooding, which is a method for preventing overfitting in supervised learning. In this case, we need to consider “how to apply flooding to GANs” and “how to set the flood level,” which are explained in the following sections.

How to apply flooding to GANs We propose to apply flooding to $L_{D}$ to avoid the overfitting of the discriminator. There are three ways to apply flooding, depending on the inserting position of the operation $h$ defined in Eq. (4),

where $b_{\,\text{real}}$ and $b_{\,\text{fake}}$ are the flood levels for adversarial losses for real and generated data, respectively, and $b_{\,\text{all}}$ is the flood level for the sum of the adversarial losses. As flooding and iFlood, the difference in the flood level and inserting position can cause performance improvement.

How to set the flood level The appropriate setting of the flood level $b_{\,\text{real}}$, $b_{\,\text{fake}}$, and $b_{\,\text{all}}$ is important. In supervised learning, because the loss at convergence is not uniquely determined but it depends on models and datasets, the flood level is a hyperparameter, and the appropriate range of it has not yet been shown. On the other hand, we propose the following hypotheses about the setting of the flood level for GANs by using property that $L_{D_{\mathrm{opt}},\mathrm{real}}$, $L_{D_{\mathrm{opt}},\mathrm{fake}}$, or the sum of the losses $L_{D_{\mathrm{opt}}}$ is uniquely determined, as summarized in Table 2.

If $L_{D_{\mathrm{opt}},\mathrm{real}}$ and $L_{D_{\mathrm{opt}},\mathrm{fake}}$ are uniquely determined, we can suppose excessively low loss for each losses. This is the inspiration for Hypothesis 1. For example, since $L_{D_{\mathrm{opt}},\mathrm{real}}$ and $L_{D_{\mathrm{opt}},\mathrm{fake}}$ are $\log 2$ for GANs with the BCE loss, $b_{\,\text{real}}$ and $b_{\,\text{fake}}$ should be set lower than $\log 2$. On the other hand, there are cases where $L_{D_{\mathrm{opt}}}$ rather than $L_{D_{\mathrm{opt}},\mathrm{real}}$ and $L_{D_{\mathrm{opt}},\mathrm{fake}}$ are uniquely determined as the hinge loss. In such cases, because $L_{D_{\mathrm{opt}}}$ is expressed as $L_{D_{\mathrm{opt}},\mathrm{real}}+L_{D_{\mathrm{opt}},\mathrm{fake}}$, we should set $b_{\,\text{real}}$ and $b_{\,\text{fake}}$ according to the sum. Moreover, the setting $b_{\,\text{all}}$ should be lower than $L_{D_{\mathrm{opt}}}$ as Hypothesis 1. This is the inspiration for Hypothesis 2.

The theorem is important in the following three points.

1. 1.

   When $e^{-b_{\,\text{real}}}+e^{-b_{\,\text{fake}}}<1$, the output of $D^{*}$ becomes a constant, which does not relate to $\mathbb{P}_{g}$ and $\mathbb{P}_{r}$, and then training will collapse.

2. 2.

   When $e^{-b_{\,\text{real}}}+e^{-b_{\,\text{fake}}}>1$ and the inequality  (10) is held, the output of $D^{*}$ is the same as Eq. (7). Moreover, if $b_{\,\text{real}}<\log 2$ and $b_{\,\text{fake}}<\log 2$ is satisfied, $e^{-b_{\,\text{real}}}+e^{-b_{\,\text{fake}}}>1$ and the inequality (10) are held where $\mathbb{P}_{g}=\mathbb{P}_{r}$. Because the goal of GANs is that the generator satisfies $\mathbb{P}_{g}=\mathbb{P}_{r}$, $b_{\,\text{real}}<\log 2$ and $b_{\,\text{fake}}<\log 2$ is required for GAN training convergence.

3. 3.

   When $e^{-b_{\,\text{real}}}+e^{-b_{\,\text{fake}}}>1$, the discriminator with a higher flood level is more difficult to satisfy the inequality (10).

The details are shown in Supplementary Section 0.B. We assume that higher flood levels increase the dangers mentioned in the third point while lower flood levels diminish the stabilizing effect. Therefore, it is necessary to investigate the appropriate setting of the flood level through experiments.

We briefly summarize the implementation below. The details are provided in Supplementary Section 0.D.

Synthetic Dataset To examine the effect of flooding for training GANs, we used the ring of 2D Gaussian dataset (2D Ring) as previous research [21, 7, 10, 34, 22]. We did five runs and evaluated the variety and quality of generated samples with ‘modes’ and ‘high quality (HQ),’ proposed in [31]. For instance, for 2D Ring, it holds that modes $\leq 8$ and HQ $\leq 1$. In order to confirm how much flooding prevents mode collapse, we consider higher modes as better performance. If modes are the same, we consider one with higher HQ as better performance.

DCGAN We used unconditional DCGAN [27] to evaluate the performance of image generation. We used CIFAR10, CIFAR100 at 32$\times$32, STL10 at 64$\times$64, CelebA at both 64$\times$64 and 128$\times$128. We conducted each experiment five times and evaluated the generated images using Fréchet Inception Distance (FID) [11].

Large model We conducted experiments using StarGAN V2 [5] to investigate the effect of flooding in the generation of larger images. We followed the author’s implementation and used CelebAHQ. We conducted each experiment five times and evaluated the generated images with FID and LPIPS [41].

First, we investigated the stabilizing effect of GAN training by flooding with a synthetic dataset and determined which flooding type in Eq. (6) is appropriate. We used the non-saturating BCE loss, which will be referred to as the BCE loss in the latter part of the paper. Note that the discussion in Section 3.2 does not use $f_{G}=\log(1-D(G(z)))$, and we can make the same arguments with the non-saturating BCE loss. To compare a case without flooding and to find the appropriate flood level, we conducted experiments on five different settings (Small, Medium, Near Opt, Opt, Over opt) with condition $b_{\,\text{real}}=b_{\,\text{fake}}$. The details of the setting are described in Supplementary Section 0.D.

Table 3 shows the average and standard deviation of the modes and HQ. Results without flooding indicate a high HQ (0.90) but low modes (4.8). This suggests that only a few of the eight Gaussian centers are accurately represented, implying that mode collapse occurred. With flooding type $L_{D,\text{flood},1}$, the results of flood levels at Small, Medium, and Near Opt show that the modes are better and HQ achieves a high value. Specifically, with flood level Medium, the generator expresses all of the Gaussian centers in four out of five runs. On the other hand, with flooding type $L_{D,\text{flood},2}$ and $L_{D,\text{flood},3}$, the results of flood level Small and Medium are poor in mode, and the result of flood level Near Opt is poor in HQ. This suggests that instance-level flooding $L_{D,\text{flood},1}$ and flood level under $L_{D_{\mathrm{opt}},\mathrm{real}}$ and $L_{D_{\mathrm{opt}},\mathrm{fake}}$ is effective in GAN training. Because without flooding the discriminator avoids taking losses below the flood level, it also shows that training becomes unstable when the loss is too low at the instance-level. It is also worth noting that the flood level Near Opt with flooding type $L_{D,\text{flood},1}$ stabilizes the training. It suggests that GAN training can progress even if the discriminator loses its ability to take low losses. As shown in Theorem 3.1, setting a larger flood level has a drawback that reduces the probability of satisfying the inequality (10). However, the experimental result suggests that preventing destabilization is more beneficial than the drawback. We can also see that the training completely collapses when the flood level $b_{\,\text{real}}$, $b_{\,\text{fake}}$, and $b_{\,\text{all}}$ takes more than $L_{D_{\mathrm{opt}},\mathrm{real}}$, $L_{D_{\mathrm{opt}},\mathrm{fake}}$, and $L_{D_{\mathrm{opt}}}$. As shown in Figure 2 (a) and (b), compared with the modes and HQ transition without flooding, the collapse with flood level Opt and Over Opt started early on. It indicates that such a configuration disrupts the GAN training, as discussed in Section 3.2. We provide analysis on the loss and gradient with flooding in Supplementary Section 0.F.

We also tested the effect of flooding with other adversarial losses: the hinge loss, the least squares loss, and the Wasserstein loss with gradient penalty (WGAN-GP). The flood level setting is provided in Table 10.

The results are shown in Table 4. It indicates that flooding with various adversarial losses and flood level Medium stabilizes the training.

In Section 4.2 we confirmed the effect of flooding with flooding type $L_{D,\text{flood},1}$, where flooding is applied to the discriminator’s loss for both real and generated data, we call it ‘two-sided flooding’. As a special case, we investigated what happens if flooding was applied only to the loss for real data (‘one-sided (real)’ flooding) or generated data (‘one-sided (fake)’ flooding), i.e., when we set only $b_{\mathrm{real}}$ or $b_{\mathrm{fake}}$, respectively. We did experiments on various adversarial losses with flood level Medium.

Table 5 shows the results. We can see that the performance with one-sided (fake) flooding is comparable to that of two-sided flooding except for the Wasserstein loss. It is noteworthy that one-sided (real) led to significant performance degradation in the BCE loss, the hinge loss, and the least-squares loss. It suggests that the discriminator overfits the generated data rather than real data with the setting. In other words, although it is difficult for the discriminator to overfit real samples drawn from the true distribution at each iteration, it is easier to overfit the generated samples from the generator whose expressiveness is low. In Figure 2 (c), we show the transition of generated samples with the BCE loss.

Next, we compared the effect of flooding and label smoothing. Label smoothing and the proposed method share similarities as stabilization techniques that do not add regularization terms while preserving the type of adversarial losses. Label smoothing calculates $L_{D,\text{real}}$ with $a=0.9$ (recommended by [28]),

Table 5 shows that the label smoothing has a positive effect only for the BCE loss and the hinge loss, and it worsens the performance for the least squares loss and the Wasserstein loss while flooding performs well for all four losses.

We provide additional experiments in Supplementary Section 0.E.

Flooding for various adversarial losses Next, we examined the effect of flooding on image generation tasks. We use various adversarial losses, DCGAN, and CelebA (128$\times$128) with flood level Medium.

We show the results in Table 6 (a) and the generated images in Figure 3. The results show that the combination of one-sided (real) flooding and the change in the type of adversarial loss further stabilizes the training. On the other hand, it seems that two-sided or one-sided (fake) is not effective in most cases.

Flooding for various datasets Next, we verified the stabilization effect of flooding regardless of the dataset. We conducted experiments on CIFAR10, CIFAR100, and STL10, with DCGAN, the BCE loss, and flood level Medium.

We show the results in Table 7. Although the difficulty of the image generation task is different among the datasets, one-sided (real) flooding is effective on various datasets. One reason why one-sided (real) flooding is effective is because it prevents the discriminator from memorization in GANs, hypothesized in Section 4.2 of [3]. In other words, if the discriminator memorizes the limited dataset, its loss would sharply drop, but flooding can prevent it.

Additional regularization terms and architectural changes can stabilize the GAN training. We examined the effect of flooding in combination with spectral normalization and gradient penalty, commonly used as improvement methods, on DCGAN and CelebA (128$\times$128), with the BCE loss. Note that we used gradient penalty with the BCE loss as ‘GAN-GP’ in [24], while we used it with the Wasserstein loss in Section 3 , which is necessary for the theoretical proof of training convergence [10].

Table 6 (b) shows the results. Even when using the gradient penalty for the BCE loss or spectral normalization alone, they could not prevent the collapse of GAN training on the dataset. On the other hand, using these regularization with flooding improves the performance, which shows the effect of flooding.

We examined the effect of flooding with other adversarial applications, such as conditional GANs [23] and domain adaptation. For the experiments of conditional GANs, we conduct experiments with conditional DCGAN (CDCGAN) and CIFAR10. To verify the effect of flooding on adversarial learning for domain adaptation, we used ADDA [35], DANN [8], and WDGRL [29]. We conducted experiments with MNIST (source domain) and MNIST-M (target domain).

Table 8 shows the result. First, the results demonstrate that flooding performs well on CDCGAN. Moreover, it shows that flooding was significantly effective on ADDA, but not on DANN and WDGRL. One possible interpretation is that the discriminator only in ADDA causes overfitting. DANN and WDGRL update the model that generates features from the source domain data during the training, whereas ADDA fixes the model. Therefore, it’s possible that in ADDA, the discriminator overfits fixed source domain features from the fixed model, but flooding prevents it. Therefore, it supports the effect of flooding in a wide range of adversarial applications to prevent overfitting and increase performance.

We verified the effect of flooding in large-scale GANs. StarGAN V2  [5] demonstrates high performance in image-to-image translation. The generator generates style codes from random latent codes or reference images, and then generates images from source images and the style codes. We tried one-sided (real) flooding because it performs better for image generation in Section 4.6.

Table 9 shows the performance. One-sided (real) flooding improved three out of four metrics. For the FID (reference) measurement where performance dropped, the variance is significantly larger than that of other measurements. Therefore, we believe its reliability is lower than the others. Figure 4 shows examples of the generated images.

While flooding demonstrates stabilization effects in the training of various GANs, we observed collapsed results with CelebA (128$\times$128) on one for five runs. Therefore, we should be aware that flooding cannot prevent all instabilities of GAN training. For example, well-known existing methods, such as gradient penalty and spectral normalization ensure Lipschitz continuity of the discriminator; however, it is beyond the scope of our approach. Therefore, flooding should be combined with other stabilization techniques appropriately.