RoNet: Rotation-oriented Continuous Image Translation

During the training, the generator takes the content features $C$ and style features $S$ as inputs, learning to generate realistic images via an adversarial loss:

where $D_{y}(\cdot)$ means the output of discriminator $D$ corresponding to the domain $y$. Notably, style features $S_{mix}$ are $S_{rot}$ or $S_{tgt}$. Concretely, we input $S_{rot}$ and $S_{tgt}$ alternately to generator for better harmonization of style encoder $E_{s}$ and rotation plane $W$ during training. In certain aspects, such training strategy helps us discover better cyclic manifold and save computation memory.

For better content representation, we use a pre-trained VGG network [75] to extract content feature maps for computing the content perceptual loss as follows:

where $\varphi_{i}(\cdot)$ denotes the features extracted from the $i$-th layer in a pre-trained VGG network [75] and $\left\|\cdot\right\|_{2}$ denotes Mean Square Error (MSE). Besides, $cl$ represents layers {$conv4\_1$, $conv5\_1$}.

Although adversarial and content losses provide generator with powerful constraints, unaligned semantic information between domains results in semantic flipping. Therefore we adapt a VSA-based loss [27] to preserve the principal objects of source domain.

where $dist(\cdot,\cdot)$ is the cosine distance. $V_{src}$ and $V_{mix}$ are hypervectors to present the semantic features of $I_{src}$ and $I_{mix}$ respectively, obtained by projecting image features in locality sensitive hashing (LSH) [76, 27].

Facilitating style encoder $E_{s}$ to capture the realistic texture, we propose a patch-based semantic style loss. In detail, the target image $I_{tgt}$ is uniformly divided into 16 patches while randomly sampling $N$ patches $B^{N}_{mix}$ from the generated image $I_{mix}$. Then cosine similarity matrix is obtained by calculating the cosine distance of patch-based latent representations from two domains, which are extracted by a pre-trained VGG encoder [75]. According to semantic similarity, we can match $N$ patches $B^{N}_{tgt}$ from $I_{tgt}$, processing analogical semantic as $B^{N}_{mix}$. Thus the same objects from different domains are matched for more accurate style learning. For example, the style of trees in $I_{mix}$ is learned from that in $I_{tgt}$. In this work, we adapt a DINO-ViT model [77] (a Vision Transformer model that has been pre-trained in a self-supervised manner) to obtain the $[CLS]$ token, which contains the semantic style information of the image. Therefore we can ensure style capture by minimizing the $[CLS]$ token distances as shown in Figure 6.

where $e^{L}_{[CLS]}$ denotes the last layer $[CLS]$ token of DINO-ViT [77].

In order to make style features flow into the cyclic manifold as much as possible. We conduct a loss to restrict the $S_{rot}$ and $S_{tgt}$, ensuring the style features can be rotated from the source domain to the target domain.

Notably, this loss term only work when the input of generator is $S_{rot}$, which helps to make encoder and rotation plane adapt each other.

To further guarantee that the generator $G$ learns to preserve the original domain-invariant characteristics, we build a reconstruction loss:

Therefore, the network is trained by minimizing the loss function defined as:

where $\lambda_{adv}$, $\lambda_{con}$, $\lambda_{VSA}$, $\lambda_{rec}$, $\lambda_{mse}$, $\lambda_{sty}$ are the hyper-parameters to balance each item.

During training, we use the ADAM solver [78] with $\beta_{1}=0.0$ and $\beta_{2}=0.99$, and a learning rate of 0.0001 for optimization. Besides, we empirically set the hyperparameters $\lambda_{adv}$, $\lambda_{con}$, $\lambda_{VSA}$, $\lambda_{rec}$, $\lambda_{mse}$ and $\lambda_{sty}$ as 1, 1, 3, 50, 150 and 1 respectively, making the losses balanced. All the experiments are conducted under the environment of Python 3.7.3 and PyTorch 1.7.1 on an Ubuntu 18.04 system with one single 32G Tesla-V00 GPU. In the training stage, all images are randomly cropped into 512 $\times$ 512, while in the inference stage, any image size is supported.

Our proposed framework is allowed to be applied in different scenes, all experiments are conducted on multiple datasets as follows.

1. 1.

   Season Album: As for seasonal image translation, we collect 4000 training images and 1000 testing images for each season from flickr.com. The resolution of all seasonal images is greater than 1024, for keeping the details of seasonal characteristics such as the color of leaves, the texture of snow, etc.

2. 2.

   Comic Face: We obtain real and comic face dataset from kaggle.com, which is used to finish real face stylization. There are 1500 real identities and 1500 anime faces for training, and 500 images for testing.

3. 3.

   Waymo Open Dataset [79] : There is high-resolution sensor data collected by autonomous vehicles operated by the Waymo Driver in a wide variety of conditions. The scenes of dateset are selected from both suburban and urban area at different moments of the day. The dataset is currently released to making advancements in machine perception and self-driving technology. As for timeshift task, we split clear images into four domains {day, dusk, night, dawn} according to Waymo image labels, obtaining train / test sets of 27272 / 7682 images.

4. 4.

   Iphone2dslr Flowers [3] : The dataset is used to map iPhone images with large depth of field to DSLR images with shallow depth of field. There are 1812 / 569 iPhone images and 3325 / 480 DSLR images for training / testing.

We compared our methods with several state-of-the-art methods as follows, which can accomplish the continuous image-to-image translation in different interpolation ways.

1. 1.

   StarGAN v2 [19] is a state-of-the-art multi-domain translation network. It can map an input image to multiple defined domains with a single model. We train the model with the public codes released by the authors and use its disentangled style code to enhance the continuous effects with linear interpolation.

2. 2.

   DLOW [17] realizes continuous translation by generating a continuous sequence of intermediate labels between two domains. In other words, it is a method based on interpolated labels.

3. 3.

   DRIT [6] is able to generate diverse results within a certain domain, but not suitable for multi-domain translation directly. Thus we train the model between every two domains and apply interpolation to the models to obtain the continuous results.

4. 4.

   Fast Photo Style [7] is a style transfer method based on disentanglement learning and can generate continuous transfer results via linear interpolation.

5. 5.

   VSAIT [27] is a paradigm for image-to-image translation by setting VSA-based constraints on adversarial learning, achieving continuous image generation by applying the interpolation to the models.

6. 6.

   CoMoGAN [71] achieves cyclic continuous translation with the guidance of physics-inspired models, but is limited in scenarios without decent physical models, such as season transition.

7. 7.

   DiffuseIT [80] is a score-based model to accomplish image translation by introducing a loss function to control the diffusion process, but there are instances of content and style leakage in the generated results.

We choose the following quantitative evaluation metrics to demonstrate the effectiveness of our proposed framework.

1. 1.

   Learned Perceptual Image Patch Similarity (LPIPS) is based on the VGG [75] and AlexNet [81] network architectures, evaluating of the distance between image patches. Higher means further different, while lower means more similar.

2. 2.

   Fréchet Inception Distance (FID) [82] is a metric that compares the distribution of generated images with the distribution of a set of real images, by calculating the distance between feature vectors of real and generated images. The FID is the current standard metric for assessing the quality of generative models.

3. 3.

   Kernel Inception Distance (KID) [83] is able to calculate the squared Maximum Mean Discrepancy (MMD) between the Inception representations of the real and generated images, via a polynomial kernel. Similar to the FID [82], lower values indicate closer distances between the distributions of generated and real data.

Fig.7 presents the continuous translation results with high definition. Please zoom in for more realistic details. In nature, most plants turn into dark yellow when autumn comes. RoNet captures the smooth variation successfully with the help of the in-plane rotation. For typical forest scene as the first row, the complex texture is quite challenging for the net to learn, easily causing artifacts or blur. It is observed that RoNet preserves the texture well because there is the patch-based semantic style loss. In the penultimate row, the plants changes with seasons, while the river keeps similar appearance across seasons, which is agree with natural order. Furthermore, RoNet has the ability of cyclic continuous translation with a single input. We present raw results in Fig.1 which are also the materials for the corresponding time-lapse demo in the supplementary material.

We compare RoNet with leading approaches in closely related fields, including multi-domain translation, continuous translation based on linear interpolation, and translation based on disentangled representations (StarGAN v2 [19], DLOW [17], DRIT [6], Fast Photo Style [7], VSAIT [27], DiffuseIT [80]). Note that all the approaches are trained and tested under the same protocol. The comparison is exhibited in Fig.8. For StarGAN v2 [19], we provide the style representations of different target domains to implement the multi-domain translation. Thus the 2nd, 5th, 8th and last column of StarGAN v2 [19] is guided by certain domain styles, while the rest are the corresponding interpolation results over the style vectors. For the other four rows, models are trained between Spring and Winter as they are essentially point-to-point approaches, but applied with different interpolation schemes: DLOW [17] with label interpolation, Fast Photo Style [7] and DiffuseIT [80] with representation interpolation, DNI-DRIT [6] and DNI-VSAIT [27] with model interpolation. It is observed that RoNet achieves the most appealing results in terms of both visual quality and continuity.

In order to evaluate the performance of our model objectively, we report the quantitative metrics of different approaches in Table I. Three metrics are employed to evaluate these approaches in various views. LPIPS [84] evaluates the structure difference between source images and generated images, FID [82] and KID [83] measures the distance between the distribution of the target images and the generated images. For all the three metrics, the smaller the better. Note that all contrastive experiments are conducted with test set in datasets. It is observed that RoNet achieves the lowest performance in all metrics, verifying its superiority over the others.

To analyse the contribution of each loss function elaborately, we conduct the ablation studies and show the results in Tab.II. The baseline is trained merely by the adversarial loss, then we add each loss function gradually. If we first add the patch-based semantic style loss, the model performance declines much. The reason is that these two loss functions focus more on visual reality but are insufficient for model convergence. With the contribution of the content loss, the reconstruction loss and the MSE loss, the performance increases significantly. Finally, the full model F achieves the best results. In the paper, we report the importance of each loss function as presented in Tab.II. Here, visual effects of each loss function are shown in Fig.9. It is observed that the sky artifacts decrease and semantic style becomes more realistic. In addition, we remove the rotation module to test its impact on the model’s performance. The results are shown in Tab.II and Fig.9. The findings demonstrate that the rotation module significantly enhances the model’s ability to learn vibrant styles, even without reference style images.

We apply different interpolation schemes to the state-of-the-art methods: DLOW [17] with label interpolation, StarGAN v2 [19] and Fast Photo Style [7] with representation interpolation, DNI-DRIT [6] and DNI-VSAIT [27] with model interpolation. As shown in Fig.10, we set the spring image as input source and the winter image as target style. It is observed that StarGAN v2 [19], DiffuseIT [80] and Fast Photo Style [7] suffer from content leak, DNI-DRIT [6] and DNI-VSAIT [27] exhibit serious artifacts. However, linear interpolation is not always valid, these methods should generate intermediate results of the summer and autumn seasons. DLOW [17] and RoNet achieve the full seasonal circulation without the target image, but DLOW [17] obtains more style leak.

Due to the advantageous features of convolutional neural networks, RoNet is capable of producing a plethora of images with varying scales. As illustrated in Fig.11, the images generated by RoNet possess a resolution of $1024\times 1600$ pixels, which vividly exposes the intricate texture details of leaves.

Recently, an unsupervised generation network named CoMoGAN [71] has been proposed to learn non-linear continuous translations. Despite the unsupervised training manner, CoMoGAN relies on physics-inspired models to guide the learning process. Taking the cyclic translation task of “day to any time” as an example, CoMoGAN first renders a daytime image with a color-based model to obtain the images at any time of a day, and then use the series of rendered data as supervision. It makes CoMoGAN stuck in the limitation of seeking the proper physical model. When confronting more challenging tasks such as season transition, CoMoGAN is out of work since it is hard to find a capable physical model for data rendering. However, our proposed framework is not constrained by physical guidance, applying to arbitrary scene domains, such as seasonal variation shown in Fig.7 and time shifting shown in Fig.13.

We compare RoNet with several SOTA approaches (StarGAN v2 [19], DLOW [17], Fast Photo Style [7], DRIT [6], CoMoGAN [71], VSAIT [27], DiffuseIT [80]). From Fig.12 and Tab.III, it is observed that StarGAN v2 [19] and DiffuseIT [80] suffer from serious semantic distortion. Fast Photo Style [7], DLOW[17], DNI-VSAIT [27] and DNI-DRIT [6] obtain unrealistic results. Since the guidance of physics-inspired function, CoMoGAN [71] learns more color- and pixel-wise information. However, our model captures the style from data distribution, such as the street lights in the night and sunsets in the dusk.

Our proposed method is a general framework that can be applied to most natural scenes with continuous variance, including seasonal circulation and time shifting. However, there are certain special circumstances in the world that need to be considered. For example, in normal situations, the length of day is longer than that of night, dawn and dusk, as shown in Fig.18. Additionally, in the southern hemisphere, the length of spring and summer is longer than that of autumn and winter, as shown in Fig.17. It is important to account for these special circumstances to ensure the effectiveness of our method in such situations.

learning imbalanced datasets with label distribution aware margin loss [85]