Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Joint intensity–gradient guided generative modeling for colorization

  • Original Article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

This paper proposes an iterative score-based generative model for solving the automatic colorization problem. Although unsupervised learning methods have shown the capability to generate plausible color, inadequate exploration of detailed information and data dimensions still limit the performance of the colorization model. Considering that the number of samples in score-based generative model has influence on estimating the target gradients and the gradient map possesses important latent information of the image, the inference process of the generative modeling is conducted in joint intensity–gradient domain for colorization. Specifically, a set of intensity–gradient formed high-dimensional tensors are trained, via the score matching, to attain the gradient of data distribution in joint intensity–gradient domain. As the score function is determined, data samples are generated by means of annealed Langevin dynamics, forming an iterative colorization procedure. Furthermore, the joint intensity–gradient constraint in data-fidelity term is proposed to limit the degree of freedom within generative model at the iterative colorization stage, thus being conducive to edge-preserving colorization effect. Experimental results conveyed the remarkable performance and diversity of our proposed method.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

References

  1. Baig, M.H., Torresani, L.: Multiple hypothesis colorization and its application to image compression. Comput. Vis. Image Understand. pp. 111–123 (2017).

  2. Liu, Q., Li, S., Xiong, J., Qin, B.: WpmDecolor: weighted projection maximum solver for contrast-preserving decolorization. Vis. Comput. 35(2), 205–221 (2019)

    Article  Google Scholar 

  3. Zhang, X., Liu, S.: Contrast preserving image decolorization combining global features and local semantic features. Vis. Comput. 34, 1099–1108 (2018)

    Article  Google Scholar 

  4. Wu, J., Shen, X., Liu, L.: Interactive two-scale color-to-gray. Vis. Comput. 28, 723–731 (2012)

    Article  Google Scholar 

  5. Frans, K.: Outline colorization through tandem adversarial networks. arXiv:1704.08834 (2017).

  6. Qu, Y., Wong, T.T., Heng, P.A.: Manga colorization. ACM Trans. Graph. 25(3), 1214–1220 (2006)

    Article  Google Scholar 

  7. Limmer, M., Lensch, H.P.: Infrared colorization using deep convolutional neural networks. In: Proceedings of IEEE International Conference on Machine Learning and Application (ICMLA), pp. 61–68 (2016).

  8. Guo, J., Pan, Z., Lei, B., Ding, C.: Automatic color correction for multisource remote sensing images with Wasserstein CNN. Remote Sensing 9(5), 483 (2017)

    Article  Google Scholar 

  9. Lei, C., Chen, Q.: Fully automatic video colorization with self-regularization and diversity. In: Proceedings of IEEE Conference Computer Visual Pattern Recognition, pp. 3753–3761 (2019).

  10. Isola, P., Zhu, J.Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 1125–1134 (2017).

  11. Levin, A., Lischinski, D., Weiss, Y.: Colorization using optimization. In: ACM SIGGRAPH 2004 Papers, Vol. 23, no. 3, pp. 689–694, (2004).

  12. Welsh, T., Ashikhmin, M., Mueller, K.: Transferring color to greyscale images. ACM Trans. Graph. 21(3), 277–280 (2002)

    Article  Google Scholar 

  13. Manjunatha, V., Iyyer, M., Boyd-Graber, J., Davis, L.: Learning to color from language. In: North American Chapter of the Association for Computational Linguistics (NAACL), pp. 764–769 (2018).

  14. Bahng, H., Yoo, S., Cho, W., Keetae Park, D., Wu, Z., Ma, X., Choo, J.: Coloring with words: Guiding image colorization through text-based palette generation. In: Proceedings of Euro Conference Computer Vision, pp. 431–447 (2018).

  15. Lee, J., Kim, E., Lee, Y., Kim, D., Chang, J., Choo, J.: Reference-based sketch image colorization using augmented-self reference and dense semantic correspondence. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 5801–5810 (2020).

  16. Li, B., Lai, Y.K., John, M., Rosin, P.L.: Automatic example-based image colorization using location-aware cross-scale matching. IEEE Trans. Image Process. 28(9), 4606–4619 (2019)

    Article  MathSciNet  MATH  Google Scholar 

  17. Xu, Z., Wang, T., Fang, F., Sheng, Y., Zhang, G.: Stylization-based architecture for fast deep exemplar colorization. In: Proceedings of IEEE Conference Computer Vision Pattern Recognition pp. 9363–9372 (2020).

  18. Cao, Y., Zhou, Z., Zhang, W., Yu, Y.: Unsupervised diverse colorization via generative adversarial networks. In: Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery Databases. Springer, Cham, pp. 151–166 (2017).

  19. Charpiat, G., Hofmann, M., Schölkopf, B.: Automatic image colorization via multimodal predictions. In: Proceedings of European Conference Computer Vision. Springer, Berlin, pp. 126–139 (2008).

  20. Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: IEEE International Conference on Computer Vision, pp. 415–423 (2015).

  21. Guadarrama, S., Dahl, R., Bieber, D., Norouzi, M., Shlens, J., Murphy, K.: Pixcolor: Pixel recursive colorization. In: British Machine Vision Conference (BMVC) (2017).

  22. Morimotoand, Y., Taguchii, Y., Naemura, T.: Automatic colorization of grayscale images using multiple images on the web. In: Proceedings of SIGGRAPH, pp. 59–59 (2009).

  23. Su, J.W., Chu, H.K., Huang, J.B. (2020) Instance-aware image colorization. In: Proceedings of IEEE Conference Computer Vision Pattern Recognition, pp. 7968–7977 (2020).

  24. Zhang, R., Isola, P., Efros, A.A.: Colorful image colorization. In: Proceedings of European Conference Computer Vision, pp. 649–666 (2016).

  25. Vitoria, P., Raad, L., Ballester, C.: ChromaGAN: adversarial image colorization with semantic class distribution. In: IEEE Winter Conference on Applications of Computer Vision, pp. 2445–2454 (2020).

  26. Messaoud, S., Forsyth, D., Schwing, A.G.: Structural consistency and controllability for diverse colorization. In: Proceedings of European Conference Computer Vision, pp. 596–612 (2018).

  27. Larsson, G., Maire, M., Shakhnarovich, G.: Learning representations for automatic colorization. In: Proceedings of European Conference Computer Vision, pp. 577–593 (2016).

  28. Deshpande, A., Lu, J., Yeh, M.C., Chong, M.J., Forsyth, D.: Learning diverse image colorization. In: Proceedings of IEEE Conference Computer Vision and Pattern Recognition, pp. 6837–6845 (2017).

  29. Yoo, S., Bahng, H., Chung, S., Lee, J., Chang, J., Choo, J.: Coloring with limited data: Few-shot colorization via memory augmented networks. In: Proceedings of IEEE Conference Computer Vision Pattern Recognition, pp. 11283–11292 (2019).

  30. Anwar, S., Tahir, M., Li, C., Mian, A., Khan, F.S., Muzaffar, A.W.: Image colorization: a survey and dataset. arXiv preprint arXiv:2008.10774 (2020).

  31. Yatziv, L., Sapiro, G.: Fast image and video colorization using chrominance blending. IEEE Trans. Image Process. 15(5), 1120–1129 (2006)

    Article  Google Scholar 

  32. Sheikh, H.R., Bovik, A.C.; Information theoretic approaches to image quality assessment. In: Handbook of Image and Video Processing. Academic Press, pp. 975–989 (2005).

  33. Gong, Y. Sbalzarini, I.F.: Image enhancement by gradient distribution specification. In: Proceedings of Asian Conference Computer Vision, pp. 47–62 (2014).

  34. Sun, Z., Feng, W., Zhao, Q., Huang, L.: Brightness preserving image enhancement based on a gradient and intensity histogram. J. Electron. Imag. 24(5), 053006–053006 (2015)

    Article  Google Scholar 

  35. Mi, Z., Liang, Z., Wang, Y., Fu, X., Chen, Z.: Multi-scale gradient domain underwater image enhancement. In: Proceedings of OCEANS-MTS/IEEE Kobe Techno-Oceans (OTO), pp. 1–5 (2018).

  36. Wang, H., Chen, Y., Fang, T., Tyan, J., Ahuja, N.: Gradient adaptive image restoration and enhancement. In: Proceedings of IEEE International Conference on Image Processing, pp. 2893–2896 (2006).

  37. Cho, T.S., Zitnick, C.L., Joshi, N., Kang, S.B., Szeliski, R., Freeman, W.T.: Image restoration by matching gradient distributions. IEEE. Trans. Pattern Anal. Mach. Intell. 34(4), 683–694 (2012)

    Article  Google Scholar 

  38. Petrovic, V.S., Xydeas, C.S.: Gradient-based multiresolution image fusion. IEEE Trans. Image Process. 13(2), 228–237 (2004)

    Article  MATH  Google Scholar 

  39. Pan, J., Hu, Z., Su, Z., Yang, M.H.: L0-regularized intensity and gradient prior for deblurring text images and beyond. IEEE. Trans. Pattern Anal. Mach. Intell. 39(2), 342–355 (2016)

    Article  Google Scholar 

  40. Gooch, A.A., Olsen, S.C., Tumblin, J., Gooch, B.: Color2gray: Salience-preserving color removal. ACM Trans. Graph. 24(3), 634–639 (2005)

    Article  Google Scholar 

  41. Song, Y., Ermon, S.: Generative modeling by estimating gradients of the data distribution. In: Proceedings of Advances on Neural Information Processing System, pp. 11918–11930 (2019).

  42. Vincent, P.: A connection between score matching and denoising autoencoders. Neural Comput. 23(7), 1661–1674 (2011)

    Article  MathSciNet  MATH  Google Scholar 

  43. Goodfellow, I., Abadie, J.P., Mirza, M., Xu, B., Farley, D.W., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial nets. In Proceedings of Advances in Neural Information Processing System, pp. 2672–2680 (2014).

  44. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (ICLR), (2014)

  45. Kingma, D.P., Welling, M.: Auto-encoding variational bayes. In: International Conference on Learning Representations (2014).

  46. Zhou, J., Hong, K., Deng, T., Wang, Y., Liu, Q.: Progressive colorization via iterative generative models. IEEE Signal Process. Letter 27, 2054–2058 (2020)

    Article  Google Scholar 

  47. Bishop, C.M.: Mixture density networks. Aston University, Birmingham, U.K., Tech. Rep. NCRG/94/004 (1994).

  48. Liu, Q., Yang, Q., Cheng, H., Wang, S., Zhang, M., Liang, D.: Highly undersampled magnetic resonance imaging reconstruction using autoencoding priors. Magn. Reson. Med. 83(1), 322–336 (2020)

    Article  Google Scholar 

  49. Zhang, L., Wu, X., Buades, A., Li, X.: Color demosaicking by local directional interpolation and nonlocal adaptive thresholding. J. Electron. Imag. 20(2), 023016 (2011)

    Article  Google Scholar 

  50. Jayaram, V., Thickstun, J.: Source separation with deep generative priors. arXiv preprint arXiv:2002.07942 (2020).

  51. Pérez, P., Gangnet, M., Blake, A.: Poisson image editing. In: ACM SIGGRAPH, pp. 313–318 (2003).

  52. Yu, F., Seff, A., Zhang, Y., Song, S., Funkhouser, T., Xiao, J.: LSUN: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 (2015).

  53. Caesar, H., Uijlings, J., Ferrari, V.: Coco-stuff: Thing and stuff classes in context. In: Proceedings of IEEE Conference Computer Vision Pattern Recognition, pp. 1209–1218 (2018).

  54. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, A., Karpathy, Z., Khosla, A., Bernstein, M., Berg, A.C., Li, F.: ImageNet large scale visual recognition challenge. Int. J. Comput. Vis. 115(3), 211–252 (2015)

    Article  MathSciNet  Google Scholar 

  55. Zhao, J., Liu, L., Snoek, C.G.M., Han, J., Shao, L.: Pixel-level semantics guided image colorization, arXiv preprint arXiv:1808.01597 (2018).

  56. Iizuka, S., Simo-Serra, E., Ishikawa, H.: Let there be color! Joint end-to-end learning of global and local image priors for automatic image colorization with simultaneous classification. ACM Trans. Graph. 35(4), 1–11 (2016)

    Article  Google Scholar 

  57. Liu, Q., Leung, H.: Variable augmented neural network for decolorization and multi-exposure fusion. Inf. Fusion 46, 114–127 (2019)

    Article  Google Scholar 

  58. Irony, R., Cohen-Or, D., Lischinski, D.: Colorization by example. In: Proceedings of Euro graphics Symposium Rendering, vol. 2, pp. 201–210 (2005)

  59. Block, A., Mroueh, Y., Rakhlin, A.: Generative modeling with denoising auto-encoders and Langevin sampling. arXiv preprint arXiv:2002.00107 (2020).

  60. Narayanan, H., Mitter, S.: Sample complexity of testing the manifold hypothesis. In: Proceedings of Advance Neural Information Processing System (2010).

  61. Rifai, S., Dauphin, Y., Vincent, P., Bengio, Y., Muller, X.: The manifold tangent classifier. In: Proceedings of Advance Neural Information Processing System (2011).

  62. Antic, J.: A deep learning based project for colorizing and restoring old images (and video!). https://github.com/jantic/DeOldify, 2019. Online; Accessed 16 Oct 2019.

Download references

Acknowledgements

The authors sincerely thank the anonymous reviewers for their valuable comments and constructive suggestions that are very helpful in the improvement of this paper. This work was supported by National Natural Science Foundation of China (61871206, 61601450).

Author information

Authors and Affiliations

  1. School of Economics, Jiangxi University of Finance and Economics, Nanchang, 330013, China

    Kuan Xiong & Weidong Liao

  2. Department of Electronic Information Engineering, Nanchang University, Nanchang, 330031, China

    Kai Hong, Jin Li, Wanyun Li & Qiegen Liu

Authors
  1. Kuan Xiong
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Kai Hong
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jin Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Wanyun Li
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Weidong Liao
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Qiegen Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Qiegen Liu.

Ethics declarations

Conflict of interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiong, K., Hong, K., Li, J. et al. Joint intensity–gradient guided generative modeling for colorization. Vis Comput 39, 6537–6552 (2023). https://doi.org/10.1007/s00371-022-02747-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02747-0

Keywords

Search

Navigation