Multi-resolution continuous normalizing flows

Abstract

Recent work has shown that Neural Ordinary Differential Equations (ODEs) can serve as generative models of images using the perspective of Continuous Normalizing Flows (CNFs). Such models offer exact likelihood calculation, and invertible generation/density estimation. In this work we introduce a Multi-Resolution variant of such models (MRCNF), by characterizing the conditional distribution over the additional information required to generate a fine image that is consistent with the coarse image. We introduce a transformation between resolutions that allows for no change in the log likelihood. We show that this approach yields comparable likelihood values for various image datasets, with improved performance at higher resolutions, with fewer parameters, using only one GPU. Further, we examine the out-of-distribution properties of MRCNFs, and find that they are similar to those of other likelihood-based generative models.

Appendices

Appendix A Full Table 1

Table 5 presents the full version of Table 1 including other results relevant to the conclusion but not mentioned in the main paper for brevity.

Table 5 Unconditional image generation metrics (lower is better in all cases): parameters in the model, bits-per-dimension, time (in hours)

Appendix B Qualitative samples

Here we present qualitative examples of our method for the datasets of MNIST and CIFAR10.

Fig. 6

Generated samples from MNIST

Fig. 7

Generated samples from CIFAR10

Appendix C Simple example of density estimation

For example, if we use Euler method as our ODE solver, for density estimation (2) reduces to:

$$\begin{aligned} {\textbf{v}}(t_1) = {\textbf{v}}(t_0) + (t_1 - t_0)f_s({\textbf{v}}(t_0), t_0 \mid {\textbf{c}}) \end{aligned}$$

(C1)

where $f_s$ is a neural network, $t_0$ represents the "time" at which the state is image ${\textbf{x}}$, and $t_1$ is when the state is noise ${\textbf{z}}$. We start at scale S with an image sample ${\textbf{x}}_S$, and assume $t_0$ and $t_1$ are 0 and 1 respectively:

$$\begin{aligned} {\left\{ \begin{array}{ll} &{}{\textbf{z}}_S = {\textbf{x}}_S + f_S({\textbf{x}}_S,\ t_0 \mid {\textbf{x}}_{S-1})\\ &{}{\textbf{z}}_{S-1} = {\textbf{x}}_{S-1} + f_{S-1}({\textbf{x}}_{S-1},\ t_0 \mid {\textbf{x}}_{S-2})\\ &{}\vdots \\ &{}{\textbf{z}}_1 = {\textbf{x}}_1 + f_1({\textbf{x}}_1,\ t_0 \mid {\textbf{x}}_0)\\ &{}{\textbf{z}}_0 = {\textbf{x}}_0 + f_0({\textbf{x}}_0,\ t_0) \end{array}\right. } \end{aligned}$$

(C2)

Appendix D Simple example of generation

For example, if we use Euler method as our ODE solver, for generation (2) reduces to:

$$\begin{aligned} {\textbf{v}}(t_0) = {\textbf{v}}(t_1) + (t_0 - t_1)f_s({\textbf{v}}(t_1), t_1 \mid {\textbf{c}}) \end{aligned}$$

(D3)

i.e. the state is integrated backwards from $t_1$ (i.e. ${\textbf{z}}_s$) to $t_0$ (i.e. ${\textbf{x}}_s$). We start at scale 0 with a noise sample ${\textbf{z}}_0$, and assume $t_0$ and $t_1$ are 0 and 1 respectively:

$$\begin{aligned} {\left\{ \begin{array}{ll} &{}{\textbf{x}}_0 = {\textbf{z}}_0 - f_0({\textbf{z}}_0,\ t_1)\\ &{}{\textbf{x}}_1 = {\textbf{z}}_1 - f_1({\textbf{z}}_1,\ t_1 \mid {\textbf{x}}_0)\\ &{}\vdots \\ &{}{\textbf{x}}_{S-1} = {\textbf{z}}_{S-1} - f_{S-1}({\textbf{z}}_{S-1},\ t_1 \mid {\textbf{x}}_{S-2})\\ &{}{\textbf{x}}_S = {\textbf{z}}_S - f_S({\textbf{z}}_{S},\ t_1 \mid {\textbf{x}}_{S-1}) \end{array}\right. } \end{aligned}$$

(D4)

Appendix E Models

We used the same neural network architecture as in RNODE [9]. The CNF at each resolution consists of a stack of bl blocks of a 4-layer deep convolutional network comprised of 3x3 kernels and softplus activation functions, with 64 hidden dimensions, and time t concatenated to the spatial input. In addition, except at the coarsest resolution, the immediate coarser image is also concatenated with the state. The integration time of each piece is [0, 1]. The number of blocks bl and the corresponding total number of parameters are given in Table 6.

Table 6 Number of parameters for different models with different total number of resolutions (res), and the number of channels (ch) and number of blocks (bl) per resolution

Appendix F Gradient norm

In order to avoid exploding gradients, We clipped the norm of the gradients [94] by a maximum value of 100.0. In case of using adversarial loss, we first clip the gradients provided by the adversarial loss by 50.0, sum up the gradients provided by the log-likelihood loss, and then clip the summed gradients by 100.0.

Appendix G 8-bit to uniform

The change-of-variables formula gives the change in probability due to the transformation of ${\textbf{u}}$ to ${\textbf{v}}$:

$$\begin{aligned} \log p({\textbf{u}}) = \log p({\textbf{v}}) + \log \left| \det \frac{{\textrm{d}}{\textbf{v}}}{{\textrm{d}}{\textbf{u}}}\right| \end{aligned}$$

Specifically, the change of variables from an 8-bit image to an image with pixel values in range [0, 1] is:

$$\begin{aligned}&{\textbf{b}}_S^{(p)} = \frac{{\textbf{a}}_S^{(p)}}{256}\\&\implies \log p({\textbf{a}}_S) = \log p({\textbf{b}}_S) + \log \left| \det \frac{{\textrm{d}}{\textbf{b}}}{{\textrm{d}}{\textbf{a}}}\right| \\&\implies \log p({\textbf{a}}_S) = \log p({\textbf{b}}_S) + \log \left( \frac{1}{256}\right) ^{D_S} \\&\implies \log p({\textbf{a}}_S) = \log p({\textbf{b}}_S) - D_S \log 256\\ \implies&\text {bpd}({\textbf{a}}_S) = \frac{-\log p({\textbf{a}}_S)}{D_S \log 2} \\&= \frac{-(\log p({\textbf{b}}_S) - D_S \log 256)}{D_S \log 2} \\&= \frac{-\log p({\textbf{b}}_S)}{D_S \log 2} + \frac{\log 256}{\log 2}\\&= \text {bpd}({\textbf{x}}) + 8 \end{aligned}$$

where $\text {bpd}({\textbf{x}})$ is given from (17).

Appendix H FID v/s Temperature

Table 7 lists the FID values of generated images from MRCNF models trained on CIFAR10, with different temperature settings on the Gaussian.

Table 7 FID v/s temperature for MRCNF models trained on CIFAR10