research-article

Deep Learning for Anime Style Transfer

Authors:

Wei-Cheng Chang,

Zen-Chung ShihAuthors Info & Claims

ICAIP '19: Proceedings of the 2019 3rd International Conference on Advances in Image Processing

Pages 139 - 143

https://doi.org/10.1145/3373419.3373433

Published: 24 January 2020 Publication History

Abstract

Some artificial systems based on a deep neural network create artistic images of high perceptual quality. However, it is usually suitable for use in abstract styles. The performances of existing style transfer algorithms on anime style are not very satisfactory, because it is either not sufficiently stylized or distorted severely in comic characters' domain. In this paper, we propose a novel anime style transfer algorithm using deep neural network, which treats foreground and background differently. Moreover, our method also could transfer the style for video with a style image. We combine semantic segmentation and spatial control to transfer the specified style to the specified area. By designing the initial image and the loss function. Users could adjust the feature weights of different regions to maintain the artistic conception of the target style, and combine optical flow to ensure frame coherence in a video. Finally, some experimental results demonstrate the effectiveness of our proposed method.

References

[1]

Gatys, L. A., A. S. Ecker, and M. Bethge. A neural algorithm of artistic style. arXiv preprint arXiv:1508.06576, 2015.

[2]

Long, J., E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3431--3440, 2015.

[3]

Gatys, L. A., A. S. Ecker, M. Bethge, A. Hertzmann, and E. Shechtman. Controlling perceptual factors in neural style transfer. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.

[4]

Bako, S., Vogels, T., McWilliams, B., Meyer, M., Novák, J., Harvill, A., Sen, P., DeRose, T. and Rousselle F. 2017. Kernel-predicting convolutional networks for denoising Monte Carlo renderings. ACM Transactions on Graphics (TOG), 36(4):97.

[5]

Riegler, G., A. O. Ulusoy, and A. Geiger. OctNET: Learning deep 3d representations at high resolutions. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, volume 3, 2017.

[6]

Simonyan K. and A. Zisserman. Very deep convolutional networks for large scale image recognition. arXiv preprint arXiv:1409.1556, 2014.

[7]

Girshick. R., Fast r-CNN. arXiv preprint arXiv:1504.08083, 2015.

[8]

Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 779--788, 2016.

[9]

Suga, A., K. Fukuda, T. Takiguchi and Y. Ariki. Object recognition and segmentation using sift and graph cuts. In Pattern Recognition, 2008. ICPR 2008. 19th International Conference on, pages 1--4. IEEE, 2008.

[10]

Zeng, K., M. Zhao, C. Xiong and S. C. Zhu. From image parsing to painterly rendering. ACM Trans. Graph., 29(1):2--1, 2009.

Digital Library

[11]

Chen, L.-C., Yang, Y., Wang, J., Xu, W. and Yuille, A. L. 2016. Attention to scale: Scale-aware semantic image segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3640--3649.

[12]

Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. and Yuille, A. L.2018. DeepLAB: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs. IEEE transactions on pattern analysis and machine intelligence, 40(4):834--848, 2018.

[13]

Chen, L.-C., Barron, J. T., Papandreou, G., Murphy, K. and Yuille. A. L. 2016. Semantic image segmentation with task-specific edge detection using CNNs and a discriminatively trained domain transform. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4545--4554.

[14]

He, K., G. Gkioxari, P. Dollár, and R. Girshick. Mask r-CNN. In Computer Vision (ICCV), 2017 IEEE International Conference on, pages 2980--2988. IEEE, 2017.

[15]

Johnson, J., A. Alahi, and L. Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision, pages 694--711. Springer, 2016.

[16]

Ulyanov, D., A. Vedaldi, and V. Lempitsky. Instance normalization: The missing ingredient for fast stylization. arXiv preprint arXiv:1607.08022, 2016.

[17]

Liu, X.-C., M.-M. Cheng, Y.-K. Lai, and P. L. Rosin. Depth-aware neural style transfer. In Proceedings of the Symposium on Non-Photorealistic Animation and Rendering, page 4. ACM, 2017.

[18]

Ruder, M., A. Dosovitskiy and T. Brox. Artistic style transfer for videos. In German Conference on Pattern Recognition, pages 26--36. Springer, 2016.

[19]

Jia, Y., E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick, S. Guadarrama, and T. Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pages 675--678. ACM, 2014.

Digital Library

[20]

Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. and Zisserman A. 2012. The PASCAL Visual Object Classes Challenge 2012 (VOC2012) Results. http://www.pascal- network.org/challenges/VOC/voc2012/workshop/index.html.

Cited By

Yu HXu WTan TLiu ZShi J(2024)Prediction of drug–target binding affinity based on multi-scale feature fusionComputers in Biology and Medicine10.1016/j.compbiomed.2024.108699178:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.compbiomed.2024.108699
Gao WZhang L(2022)Semantic Segmentation of Substation Site Cloud Based on Seg-PointNetJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2022.p100426:6(1004-1012)Online publication date: 20-Nov-2022
https://doi.org/10.20965/jaciii.2022.p1004
Yang QChen WCai YLiu XLiu TWang G(2022)LWComicGAN: A Lightweight Method for Realizing Scene Animation2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC)10.1109/ITAIC54216.2022.9836461(2285-2289)Online publication date: 17-Jun-2022
https://doi.org/10.1109/ITAIC54216.2022.9836461

Index Terms

Deep Learning for Anime Style Transfer
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing
    2. Rendering
      1. Non-photorealistic rendering

Recommendations

Semantic Correspondence Guided Deep Photo Style Transfer
Advances in Multimedia Information Processing – PCM 2018
Abstract
The objective of this paper is to develop an effective photographic transfer method while preserving the semantic correspondence between the style and content images. A semantic correspondence guided deep photo style transfer algorithm is ...
Neural Policy Style Transfer
Abstract
Style Transfer has been proposed in a number of fields: fine arts, natural language processing, and fixed trajectories. We scale this concept up to control policies within a Deep Reinforcement Learning infrastructure. Each network is ...
Highlights
- Using the expressive power of Deep Q-learning Networks to encode content and style.
Adaptive Style Modulation for Artistic Style Transfer
Abstract
Arbitrary-style-per-model (ASPM) style transfer algorithms transfer arbitrary styles based on a single model. Statistics-based learning algorithms of ASPM, represented by adaptive instance normalization (AdaIN), conduct instance normalization and ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

ICAIP '19: Proceedings of the 2019 3rd International Conference on Advances in Image Processing

November 2019

232 pages

ISBN:9781450376754

DOI:10.1145/3373419

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

In-Cooperation

Southwest Jiaotong University

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 January 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

the Ministry of Science and Technology of the Republic of China, Taiwan

Conference

ICAIP 2019

ICAIP 2019: 2019 3rd International Conference on Advances in Image Processing

November 8 - 10, 2019

Chengdu, China

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
403
Total Downloads

Downloads (Last 12 months)36
Downloads (Last 6 weeks)0

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Yu HXu WTan TLiu ZShi J(2024)Prediction of drug–target binding affinity based on multi-scale feature fusionComputers in Biology and Medicine10.1016/j.compbiomed.2024.108699178:COnline publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1016/j.compbiomed.2024.108699
Gao WZhang L(2022)Semantic Segmentation of Substation Site Cloud Based on Seg-PointNetJournal of Advanced Computational Intelligence and Intelligent Informatics10.20965/jaciii.2022.p100426:6(1004-1012)Online publication date: 20-Nov-2022
https://doi.org/10.20965/jaciii.2022.p1004
Yang QChen WCai YLiu XLiu TWang G(2022)LWComicGAN: A Lightweight Method for Realizing Scene Animation2022 IEEE 10th Joint International Information Technology and Artificial Intelligence Conference (ITAIC)10.1109/ITAIC54216.2022.9836461(2285-2289)Online publication date: 17-Jun-2022
https://doi.org/10.1109/ITAIC54216.2022.9836461
Chung CHuang S(2022)Interactively transforming chinese ink paintings into realistic images using a border enhance generative adversarial networkMultimedia Tools and Applications10.1007/s11042-022-13684-482:8(11663-11696)Online publication date: 27-Aug-2022
https://dl.acm.org/doi/10.1007/s11042-022-13684-4

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents