Global-Local Feature Alignment Loss for Photorealistic Style Transfer
Pages 90 - 95
Abstract
The problem that needs to be solved for photorealistic style transfer lies in limiting the distortion of texture details of the generated image based on the typical style transfer network. Although the existing methods achieve better stylization results, they lack sufficient style information because they do not consider the feature map comprehensively enough, leading to exposure or artifacts. This article proposes a loss function based on the contrast learning method to constrain the network to extract local and global information effectively. It ensures the consistency of distribution among regional blocks generated based on anchor points and the consistency of comparison between anchor points of the resulting image and content image in their neighborhood. This ensures consistency between local and global information comparisons. To ensure that the network is simple and effective and that enough information is extracted, this article proposes a linear covariance transformation network to achieve faithful stylization by effectively fusing feature first-order statistics with second-order statistics. Experiments show that the proposed method can faithfully achieve realistic stylization and satisfying visual effects.
References
[1]
Li, X., Liu, S., Kautz, J. and Yang, M.-H. Learning linear transformations for fast image and video style transfer. City, 2019.
[2]
Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X. and Yang, M.-H. Universal style transfer via feature transforms. Advances in neural information processing systems, 30 (2017).
[3]
Li, Y., Liu, M.-Y., Li, X., Yang, M.-H. and Kautz, J. A closed-form solution to photorealistic image stylization. City, 2018.
[4]
Yoo, J., Uh, Y., Chun, S., Kang, B. and Ha, J.-W. Photorealistic style transfer via wavelet transforms. City, 2019.
[5]
Misra, D., Nalamada, T., Arasanipalai, A. U. and Hou, Q. Rotate to attend: Convolutional triplet attention module. City, 2021.
[6]
Zamir, S. W., Arora, A., Khan, S., Hayat, M., Khan, F. S., Yang, M.-H. and Shao, L. Learning enriched features for real image restoration and enhancement. Springer, City, 2020.
[7]
Chen, T., Kornblith, S., Norouzi, M. and Hinton, G. A simple framework for contrastive learning of visual representations. PMLR, City, 2020.
[8]
Oord, A. v. d., Li, Y. and Vinyals, O. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018).
[9]
Park, T., Efros, A. A., Zhang, R. and Zhu, J.-Y. Contrastive learning for unpaired image-to-image translation. Springer, City, 2020.
[10]
Huang, X. and Belongie, S. Arbitrary style transfer in real-time with adaptive instance normalization. City, 2017.
[11]
Zhang, Y., Zhang, Y. and Cai, W. Separating style and content for generalized style transfer. City, 2018.
[12]
Gharbi, M., Chen, J., Barron, J. T., Hasinoff, S. W. and Durand, F. Deep bilateral learning for real-time image enhancement. ACM Transactions on Graphics (TOG), 36, 4 (2017), 1-12.
[13]
Gatys, L. A., Ecker, A. S., Bethge, M., Hertzmann, A. and Shechtman, E. Controlling perceptual factors in neural style transfer. City, 2017.
[14]
Sheng, L., Lin, Z., Shao, J. and Wang, X. Avatar-net: Multi-scale zero-shot style transfer by feature decoration. City, 2018.
[15]
Luan, F., Paris, S., Shechtman, E. and Bala, K. Deep photo style transfer. City, 2017.
[16]
An, J., Xiong, H., Luo, J., Huan, J. and Ma, J. Fast universal style transfer for artistic and photorealistic rendering. arXiv preprint arXiv:1907.03118 (2019).
[17]
Chiu, T.-Y. and Gurari, D. Photowct2: Compact autoencoder for photorealistic style transfer resulting from blockwise training and skip connections of high-frequency residuals. City, 2022.
[18]
Chuang, C.-Y., Robinson, J., Lin, Y.-C., Torralba, A. and Jegelka, S. Debiased contrastive learning. Advances in neural information processing systems, 33 (2020), 8765-8775.
[19]
Robinson, J., Chuang, C.-Y., Sra, S. and Jegelka, S. Contrastive learning with hard negative samples. arXiv preprint arXiv:2010.04592 (2020).
[20]
Wu, Z., Zhu, Z., Du, J. and Bai, X. CCPL: Contrastive Coherence Preserving Loss for Versatile Style Transfer. Springer, City, 2022.
[21]
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P. and Zitnick, C. L. Microsoft coco: Common objects in context. Springer, City, 2014.
[22]
Mohammad, S. and Kiritchenko, S. Wikiart emotions: An annotated dataset of emotions evoked by art. City, 2018.
[23]
Simonyan, K. and Zisserman, A. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556 (2014).
[24]
Liao, J., Yao, Y., Yuan, L., Hua, G. and Kang, S. B. Visual attribute transfer through deep image analogy. arXiv preprint arXiv:1705.01088 (2017).
[25]
He, M., Liao, J., Chen, D., Yuan, L. and Sander, P. V. Progressive color transfer with dense semantic correspondences. ACM Transactions on Graphics (TOG), 38, 2 (2019), 1-18.
[26]
Chen, S., Niu, G., Gong, C., Li, J., Yang, J. and Sugiyama, M. Large-margin contrastive learning with distance polarization regularizer. PMLR, City, 2021.
[27]
Zhan, F., Zhang, J., Yu, Y., Wu, R. and Lu, S. Modulated contrast for versatile image synthesis. City, 2022.
[28]
Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J. and Wang, Z. Photo-realistic single image super-resolution using a generative adversarial network. City, 2017.
[29]
Johnson, J., Alahi, A. and Fei-Fei, L. Perceptual losses for real-time style transfer and super-resolution. Springer, City, 2016.
[30]
Mechrez, R., Talmi, I., Shama, F. and Zelnik-Manor, L. Maintaining natural image statistics with the contextual loss. Springer, City, 2018.
[31]
Dosovitskiy, A. and Brox, T. Generating images with perceptual similarity metrics based on deep networks. Advances in neural information processing systems, 29 (2016).
[32]
Mechrez, R., Talmi, I. and Zelnik-Manor, L. The contextual loss for image transformation with non-aligned data. City, 2018.
[33]
Turlach, B. A. Bandwidth selection in kernel density estimation: A review. Citeseer, City, 1993.
[34]
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K. and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. Ieee, City, 2009.
[35]
Qu, Y., Shao, Z. and Qi, H. Non-Local Representation Based Mutual Affine-Transfer Network for Photorealistic Stylization. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44, 10 (2021), 7046-7061.
[36]
Zhang, Y., Li, M., Li, R., Jia, K. and Zhang, L. Exact feature distribution matching for arbitrary style transfer and domain generalization. City, 2022.
[37]
Xia, X., Zhang, M., Xue, T., Sun, Z., Fang, H., Kulis, B. and Chen, J. Joint bilateral learning for real-time universal photorealistic style transfer. Springer, City, 2020.
[38]
Hong, K., Jeon, S., Yang, H., Fu, J. and Byun, H. Domain-aware universal style transfer. City, 2021.
Index Terms
- Global-Local Feature Alignment Loss for Photorealistic Style Transfer
Recommendations
Efficient photorealistic style transfer with multi-order image statistics
AbstractPhotorealistic style transfer concerns rendering the style of a reference image to a content image with the restraint that the stylized image should be realistic. While the existing methods have achieved promising results, they are prone to ...
Photorealistic lighting with offset radiance transfer mapping
I3D '06: Proceedings of the 2006 symposium on Interactive 3D graphics and gamesWe propose a precomputation-based approach for the real-time rendering of scenes that include a number of complex illumination phenomena, such as radiosity and subsurface scattering, and allows interactive modification of camera and lighting parameters. ...
Local, deformable precomputed radiance transfer
Precomputed radiance transfer (PRT) captures realistic lighting effects from distant, low-frequency environmental lighting but has been limited to static models or precomputed sequences. We focus on PRT for local effects such as bumps, wrinkles, or ...
Comments
Information & Contributors
Information
Published In
December 2022
365 pages
ISBN:9781450398039
DOI:10.1145/3579895
Copyright © 2022 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 April 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
ICNCC 2022
ICNCC 2022: 2022 The 11th International Conference on Networks, Communication and Computing
December 9 - 11, 2022
Beijing, China
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 41Total Downloads
- Downloads (Last 12 months)22
- Downloads (Last 6 weeks)1
Reflects downloads up to 26 Sep 2024
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format