Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

Fast continuous patch-based artistic style transfer for videos

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

Convolutional neural network-based image style transfer models often suffer from temporal inconsistency when applied to video. Although several video style transfer models have been proposed to improve temporal consistency, they often trade off processing speed, perceptual style quality, and temporal consistency. In this work, we propose a novel approach for fast continuous patch-based arbitrary video style transfer that achieves high-quality transfer results while maintaining temporal coherence. Our approach begins with stylizing the first frame as a standalone single image using patch propagation within the content activation. Subsequent frames are computed based on the key insight that optical flow field evaluated from neighboring content activations provides meaningful information to preserve temporal coherence efficiently. To address the problems introduced from optical flow stage, we additionally incorporate a correction procedure as a post-process to ensure a high-quality stylized video. Finally, we demonstrate our method can transfer arbitrary styles on a set of examples and illustrate that our approach exhibits superior performance both qualitatively and quantitatively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Explore related subjects

Discover the latest articles, news and stories from top researchers in related subjects.

Data availability

The data used to support the findings of this study are available from the corresponding author upon request.

References

  1. Gatys, L.A., Ecker, A.S., Bethge, M.: A Neural Algorithm of Artistic Style, http://arxiv.org/abs/1508.06576 (2015)

  2. Li, C., Wand, M.: Combining Markov Random fields and convolutional neural networks for image synthesis. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2479–2486. IEEE, Las Vegas, NV, USA (2016)

  3. Dumoulin, V., Shlens, J., Kudlur, M.: A Learned Representation for Artistic Style, http://arxiv.org/abs/1610.07629 (2017)

  4. Wang, G., Li, W., Zhang, L., Sun, L., Chen, P., Yu, L., Ning, X.: Encoder-X: solving unknown coefficients automatically in polynomial fitting by using an autoencoder. IEEE Trans. Neural Netw. Learn. Syst. 33, 3264–3276 (2022). https://doi.org/10.1109/TNNLS.2021.3051430

    Article  MathSciNet  Google Scholar 

  5. Ning, X., Tian, W., Yu, Z., Li, W., Bai, X., Wang, Y.: HCFNN: High-order coverage function neural network for image classification. Pattern Recognit. 131, 108873 (2022). https://doi.org/10.1016/j.patcog.2022.108873

    Article  Google Scholar 

  6. Chen, D., Yuan, L., Liao, J., Yu, N., Hua, G.: StyleBank: An explicit representation for neural image style transfer. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

  7. Johnson, J., Alahi, A., Fei-Fei, L.: Perceptual losses for real-time style transfer and super-resolution. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision—ECCV 2016, pp. 694–711. Springer, Cham (2016)

    Chapter  Google Scholar 

  8. Kotovenko, D., Sanakoyeu, A., Lang, S., Ommer, B.: Content and style disentanglement for artistic style transfer. Presented at the Proceedings of the IEEE/CVF International Conference on Computer Vision (2019)

  9. Ruder, M., Dosovitskiy, A., Brox, T.: Artistic style transfer for videos. In: Rosenhahn, B., Andres, B. (eds.) Pattern Recognition, pp. 26–36. Springer, Cham (2016)

    Chapter  Google Scholar 

  10. Chen, J., He, Y., Zhang, Y., Han, P., Du, C.: Energy-aware scheduling for dependent tasks in heterogeneous multiprocessor systems. J. Syst. Archit. 129, 102598 (2022). https://doi.org/10.1016/j.sysarc.2022.102598

    Article  Google Scholar 

  11. Chen, T.Q., Schmidt, M.: Fast patch-based style transfer of arbitrary style. http://arxiv.org/abs/1612.04337 (2016)

  12. Wu, B., Ding, Y., Dong, Q.: Fast continuous structural similarity patch based arbitrary style transfer. Appl. Sci. 9, 3304 (2019). https://doi.org/10.3390/app9163304

    Article  Google Scholar 

  13. Abualigah, L., Diabat, A., Mirjalili, S., Abd Elaziz, M., Gandomi, A.H.: The arithmetic optimization algorithm. Comput. Methods Appl. Mech. Eng. 376, 113609 (2021). https://doi.org/10.1016/j.cma.2020.113609

    Article  MathSciNet  Google Scholar 

  14. Kong, X., Deng, Y., Tang, F., Dong, W., Ma, C., Chen, Y., He, Z., Xu, C.: Exploring the temporal consistency of arbitrary style transfer: a channelwise perspective. IEEE Trans. Neural Netw. Learn. Syst. (2023). https://doi.org/10.1109/TNNLS.2022.3230084

    Article  Google Scholar 

  15. Wu, B., Dong, Q., Sun, W.: Automatic colorization of Chinese ink painting combining multi-level features and generative adversarial networks. Fractals (2023). https://doi.org/10.1142/S0218348X23401448

    Article  Google Scholar 

  16. Yu, X., Zhou, G.: Arbitrary style transfer via content consistency and style consistency. Vis. Comput. (2023). https://doi.org/10.1007/s00371-023-02855-5

    Article  Google Scholar 

  17. Ye, W., Zhu, X., Liu, Y.: Multi-semantic preserving neural style transfer based on Y channel information of image. Vis. Comput. 39, 609–623 (2023). https://doi.org/10.1007/s00371-021-02361-6

    Article  Google Scholar 

  18. Li, D., Gao, W.: Neural style transfer based on deep feature synthesis. Vis. Comput. (2022). https://doi.org/10.1007/s00371-022-02664-2

    Article  Google Scholar 

  19. Gatys, L.A., Ecker, A.S., Bethge, M., Hertzmann, A., Shechtman, E.: Controlling perceptual factors in neural style transfer. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

  20. Jing, Y., Liu, X., Ding, Y., Wang, X., Ding, E., Song, M., Wen, S.: Dynamic instance normalization for arbitrary style transfer. Proc. AAAI Conf. Artif. Intell. 34, 4369–4376 (2020). https://doi.org/10.1609/aaai.v34i04.5862

    Article  Google Scholar 

  21. Li, Y., Wang, N., Liu, J., Hou, X.: Demystifying neural style transfer. http://arxiv.org/abs/1701.01036 (2017)

  22. Sheng, L., Lin, Z., Shao, J., Wang, X.: Avatar-Net: multi-scale zero-shot style transfer by feature decoration. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8242–8250. IEEE, Salt Lake City, UT, USA (2018)

  23. Huang, X., Belongie, S.: Arbitrary style transfer in real-time with adaptive instance normalization. Presented at the Proceedings of the IEEE International Conference on Computer Vision (2017)

  24. Li, Y., Fang, C., Yang, J., Wang, Z., Lu, X., Yang, M.-H.: Universal style transfer via feature transforms. In: Advances in Neural Information Processing Systems. Curran Associates, Inc. (2017)

  25. Li, X., Liu, S., Kautz, J., Yang, M.-H.: learning linear transformations for fast image and video style transfer. Presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)

  26. Wang, C., Ning, X., Sun, L., Zhang, L., Li, W., Bai, X.: Learning discriminative features by covering local geometric space for point cloud analysis. IEEE Trans. Geosci. Remote Sens. 60, 1–15 (2022). https://doi.org/10.1109/TGRS.2022.3170493

    Article  Google Scholar 

  27. Wang, C., Wang, X., Zhang, J., Zhang, L., Bai, X., Ning, X., Zhou, J., Hancock, E.: Uncertainty estimation for stereo matching based on evidential deep learning. Pattern Recognit. 124, 108498 (2022). https://doi.org/10.1016/j.patcog.2021.108498

    Article  Google Scholar 

  28. Ulyanov, D., Vedaldi, A., Lempitsky, V.: Improved texture networks: maximizing quality and diversity in feed-forward stylization and texture synthesis. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2017)

  29. Li, C., Wand, M.: Precomputed real-time texture synthesis with Markovian generative adversarial networks. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision—ECCV 2016, pp. 702–716. Springer, Cham (2016)

    Chapter  Google Scholar 

  30. Zhang, H., Dana, K.: Multi-style generative network for real-time transfer. Presented at the Proceedings of the European Conference on Computer Vision (ECCV) Workshops (2018)

  31. Yao, Y., Ren, J., Xie, X., Liu, W., Liu, Y.-J., Wang, J.: Attention-aware multi-stroke style transfer. Presented at the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (2019)

  32. Gu, S., Chen, C., Liao, J., Yuan, L.: Arbitrary style transfer with deep feature reshuffle. Presented at the Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (2018)

  33. Li, W., Wen, L., Bian, X., Lyu, S.: Evolvement constrained adversarial learning for video style transfer. In: Jawahar, C.V., Li, H., Mori, G., Schindler, K. (eds.) Computer Vision—ACCV 2018, pp. 232–248. Springer, Cham (2019)

    Chapter  Google Scholar 

  34. Huang, H., Wang, H., Luo, W., Ma, L., Jiang, W., Zhu, X., Li, Z., Liu, W.: Real-time neural style transfer for videos. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 7044–7052. IEEE, Honolulu, HI (2017)

  35. Chen, D., Liao, J., Yuan, L., Yu, N., Hua, G.: Coherent Online Video Style Transfer. In: 2017 IEEE International Conference on Computer Vision (ICCV). pp. 1114–1123. IEEE, Venice (2017)

  36. Liu, S., Lin, T., He, D., Li, F., Wang, M., Li, X., Sun, Z., Li, Q., Ding, E.: AdaAttN: revisit attention mechanism in arbitrary neural style transfer. Presented at the Proceedings of the IEEE/CVF International Conference on Computer Vision (2021)

  37. Deng, Y., Tang, F., Dong, W., Huang, H., Ma, C., Xu, C.: arbitrary video style transfer via multi-channel correlation. Proc. AAAI Conf. Artif. Intell. 35, 1210–1217 (2021). https://doi.org/10.1609/aaai.v35i2.16208

    Article  Google Scholar 

  38. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process. 13, 600–612 (2004). https://doi.org/10.1109/TIP.2003.819861

    Article  Google Scholar 

  39. Brox, T., Bruhn, A., Papenberg, N., Weickert, J.: High accuracy optical flow estimation based on a theory for warping. In: Pajdla, T., Matas, J. (eds.) Computer Vision—ECCV 2004, pp. 25–36. Springer, Berlin (2004)

    Chapter  Google Scholar 

  40. Ruder, M., Dosovitskiy, A., Brox, T.: Artistic style transfer for videos and spherical images. Int. J. Comput. Vis. 126, 1199–1219 (2018). https://doi.org/10.1007/s11263-018-1089-z

    Article  MathSciNet  Google Scholar 

  41. Butler, D.J., Wulff, J., Stanley, G.B., Black, M.J.: A naturalistic open source movie for optical flow evaluation. In: Fitzgibbon, A., Lazebnik, S., Perona, P., Sato, Y., Schmid, C. (eds.) Computer Vision—ECCV 2012, pp. 611–625. Springer, Berlin (2012)

    Chapter  Google Scholar 

Download references

Funding

This work was supported by the MOE (Ministry of Education in China) Project of Humanities and Social Sciences (No. 22YJC760014), and the Social Science Planning Project of Shandong Province (No. 22CWYJ10).

Author information

Authors and Affiliations

  1. School of Communication, Qufu Normal University, Rizhao, 276826, China

    Bing Wu, Qingshuang Dong & Wenqing Sun

Authors
  1. Bing Wu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Qingshuang Dong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Wenqing Sun
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qingshuang Dong.

Ethics declarations

Conflict of interest

The authors declare that there is no conflict of interest regarding the publication of this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Wu, B., Dong, Q. & Sun, W. Fast continuous patch-based artistic style transfer for videos. Vis Comput 40, 6123–6136 (2024). https://doi.org/10.1007/s00371-023-03157-6

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-023-03157-6

Keywords

Search

Navigation