Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
SpringerLink
Log in

Double discriminative face super-resolution network with facial landmark heatmaps

  • Original article
  • Published:
The Visual Computer Aims and scope Submit manuscript

Abstract

At present, most of face super-resolution (SR) networks cannot balance the visual quality and the pixel accuracy. The networks with high objective index values often reconstruct too smooth images, while the networks which can restore texture information often introduce too much high-frequency noise and artifacts. Besides, some face super-resolution networks do not consider the mutual promotion between the extracting face prior knowledge part and the super-resolution reconstruction part. To solve these problems, we propose the double discriminative face super-resolution network (DDFSRNet). We propose a collaborative generator and two discriminators. Specifically, the collaborative generator, including the face super-resolution module (FSRM) and the face alignment module (FAM), can strengthen the reconstruction of facial key components, under the restriction of the perceptual similarity loss, the facial heatmap loss and double generative adversarial loss. We design the feature fusion unit (FFU) in FSRM, which integrates the facial heatmap features and SR features. FFU can use the facial landmarks to correct the face edge shape. Moreover, the double discriminators, including the facial SR discriminator (FSRD) and the facial landmark heatmap discriminator (FLHD), are used to judge whether face SR images and face heatmaps are from real data or generated data, respectively. Experiments show that the perceptual effect of our method is superior to other advanced methods on 4x reconstruction and fit the face high-resolution (HR) images as much as possible.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10

Similar content being viewed by others

Data Availability

The FFHQ dataset that supports this study is openly available at https://github.com/NVlabs/ffhq-dataset/, reference number [36]. The CelebA dataset is openly available at http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html, reference number [37].

References

  1. Shen, M., Yu, P., Wang, R., Yang, J., Xue, L.: Image super-resolution reconstruction via deep network based on multi-staged fusion. J. Image Gr. 24(8), 1258–1269 (2019)

    Google Scholar 

  2. Liu, F., Ruan, R., Ni, H., Wang, J.: Robust face super-resolution reconstruction based on sparse coding. Telecommun. Eng. 57(8), 957–962 (2017)

    Google Scholar 

  3. Cao, M., Gan, Z., Cui, Z., Li, R., Zhu, X.: Novel neighbor embedding face hallucination based on non-negative weights and 2d-pca feature. J. Electron. Inform. Technol. 4, 777–783 (2015)

    Google Scholar 

  4. Xu, W., Sun, G., Lu, X.: Super-resolution reconstruction of face images guided by pre-training network. Ind. Control Comput. 33(6), 36–38 (2020)

    Google Scholar 

  5. Li, X., Orchard, M.T.: New edge-directed interpolation. IEEE Trans. Image Process. 10(10), 1521–1527 (2001)

    Article  Google Scholar 

  6. Zhang, L., Wu, X.: An edge-guided image interpolation algorithm via directional filtering and data fusion. IEEE Trans. Image Process. 15(8), 2226–2238 (2006). https://doi.org/10.1109/TIP.2006.877407

    Article  Google Scholar 

  7. Torabi, H., Pariz, N., Karimpour, A.: Kalman filters for fractional discrete-time stochastic systems along with time-delay in the observation signal. Eur. Phys. J. Sp. Top. 225(1), 107–118 (2016)

    Article  Google Scholar 

  8. Zhang, K., Gao, X., Tao, D., Li, X.: Single image super-resolution with non-local means and steering kernel regression. IEEE Trans. Image Process. 21(11), 4544–4556 (2012). https://doi.org/10.1109/TIP.2012.2208977

    Article  MathSciNet  MATH  Google Scholar 

  9. Yang, J., Wright, J., Huang, T.S., Ma, Y.: Image super-resolution via sparse representation. IEEE Trans. Image Process. 19(11), 2861–2873 (2010)

    Article  MathSciNet  MATH  Google Scholar 

  10. Dong, C., Loy, C.C., He, K., Tang, X.: Learning a deep convolutional network for image super-resolution. In: Fleet, D., Pajdla, T., Schiele, B., Tuytelaars, T. (eds.) Computer Vision-ECCV 2014, pp. 184–199. Springer, Cham (2014)

    Chapter  Google Scholar 

  11. Ledig, C., Theis, L., Huszár, F., Caballero, J., Cunningham, A., Acosta, A., Aitken, A., Tejani, A., Totz, J., Wang, Z., Shi, W.: Photo-realistic single image super-resolution using a generative adversarial network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 105–114 (2017). https://doi.org/10.1109/CVPR.2017.19

  12. Chen, C., Gong, D., Wang, H., Li, Z., Wong, K.-Y.K.: Learning spatial attention for face super-resolution. IEEE Trans. Image Process. 30, 1219–1231 (2021). https://doi.org/10.1109/TIP.2020.3043093

    Article  Google Scholar 

  13. Dong, C., Loy, C.C., Tang, X.: Accelerating the super-resolution convolutional neural network. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (eds.) Computer Vision-ECCV 2016, pp. 391–407. Springer, Cham (2016)

    Chapter  Google Scholar 

  14. Lai, W.-S., Huang, J.-B., Ahuja, N., Yang, M.-H.: Deep laplacian pyramid networks for fast and accurate super-resolution. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5835–5843 (2017). https://doi.org/10.1109/CVPR.2017.618

  15. Wang, Z., Chen, J., Hoi, S.C.H.: Deep learning for image super-resolution: a survey. IEEE Trans. Pattern Anal. Mach. Intell. 43(10), 3365–3387 (2021). https://doi.org/10.1109/TPAMI.2020.2982166

    Article  Google Scholar 

  16. Haris, M., Shakhnarovich, G., Ukita, N.: Deep back-projection networks for super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1664–1673 (2018). https://doi.org/10.1109/CVPR.2018.00179

  17. Shi, W., Du, H., Mei, W., Ma, Z.: (sarn)spatial-wise attention residual network for image super-resolution. Vis. Comput. 37, 1569–1580 (2021). https://doi.org/10.1007/s00371-020-01903-8

    Article  Google Scholar 

  18. Wang, X., Yu, K., Wu, S., Gu, J., Liu, Y., Dong, C., Qiao, Y., Loy, C.C.: Esrgan: enhanced super-resolution generative adversarial networks. In: Leal-Taixé, L., Roth, S. (eds.) Computer Vision-ECCV 2018 Workshops, pp. 63–79. Springer, Cham (2019)

    Chapter  Google Scholar 

  19. Zhang, Y., Tian, Y., Kong, Y., Zhong, B., Fu, Y.: Residual dense network for image super-resolution. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2472–2481 (2018). https://doi.org/10.1109/CVPR.2018.00262

  20. Shang, T., Dai, Q., Zhu, S., Yang, T., Guo, Y.: Perceptual extreme super resolution network with receptive field block. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1778–1787 (2020). https://doi.org/10.1109/CVPRW50498.2020.00228

  21. Ma, T., Tian, W.: Back-projection-based progressive growing generative adversarial network for single image super-resolution. Vis. Comput. 37, 925–938 (2021). https://doi.org/10.1007/s00371-020-01843-3

    Article  Google Scholar 

  22. Chen, Y., Tai, Y., Liu, X., Shen, C., Yang, J.: Fsrnet: End-to-end learning face super-resolution with facial priors. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2492–2501 (2018). https://doi.org/10.1109/CVPR.2018.00264

  23. Bulat, A., Tzimiropoulos, G.: Super-fan: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with gans. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 109–117 (2018). https://doi.org/10.1109/CVPR.2018.00019

  24. Ma, C., Jiang, Z., Rao, Y., Lu, J., Zhou, J.: Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5568–5577 (2020). https://doi.org/10.1109/CVPR42600.2020.00561

  25. Zhang, Y., Wu, Y., Chen, L.: Msfsr: A multi-stage face super-resolution with accurate facial representation via enhanced facial boundaries. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2120–2129 (2020). https://doi.org/10.1109/CVPRW50498.2020.00260

  26. Yin, Y., Robinson, J., Zhang, Y., Fu, Y.: Joint super-resolution and alignment of tiny faces. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 12693–12700 (2020)

  27. Hu, X., Ren, W., LaMaster, J., Cao, X., Li, X., Li, Z., Menze, B., Liu, W.: Face super-resolution guided by 3d facial priors. In: European Conference on Computer Vision, pp. 763–780 (2020). Springer

  28. Chen, C., Li, X., Yang, L., Lin, X., Zhang, L., Wong, K.-Y.K.: Progressive semantic-aware style transformation for blind face restoration. In: 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11891–11900 (2021). https://doi.org/10.1109/CVPR46437.2021.01172

  29. He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–778 (2016). https://doi.org/10.1109/CVPR.2016.90

  30. Tai, Y., Yang, J., Liu, X.: Image super-resolution via deep recursive residual network. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2790–2798 (2017). https://doi.org/10.1109/CVPR.2017.298

  31. Bulat, A., Tzimiropoulos, G.: How far are we from solving the 2d amp; 3d face alignment problem? (and a dataset of 230,000 3d facial landmarks). In: 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1021–1030 (2017). https://doi.org/10.1109/ICCV.2017.116

  32. Jolicoeur-Martineau, A.: The relativistic discriminator: a key element missing from standard gan (2018). Preprint at arXiv:1807.00734

  33. Isola, P., Zhu, J.-Y., Zhou, T., Efros, A.A.: Image-to-image translation with conditional adversarial networks. In: 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5967–5976 (2017). https://doi.org/10.1109/CVPR.2017.632

  34. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 586–595 (2018)

  35. Jo, Y., Yang, S., Kim, S.J.: Investigating loss functions for extreme super-resolution. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1705–1712 (2020). https://doi.org/10.1109/CVPRW50498.2020.00220

  36. Karras, T., Laine, S., Aila, T.: A style-based generator architecture for generative adversarial networks. IEEE Trans. Pattern Anal. Mach. Intell. 43(12), 4217–4228 (2021). https://doi.org/10.1109/TPAMI.2020.2970919

    Article  Google Scholar 

  37. Liu, Z., Luo, P., Wang, X., Tang, X.: Deep learning face attributes in the wild. In: 2015 IEEE International Conference on Computer Vision (ICCV), pp. 3730–3738 (2015). https://doi.org/10.1109/ICCV.2015.425

  38. Baltrusaitis, T., Robinson, P., Morency, L.-P.: Constrained local neural fields for robust facial landmark detection in the wild. In: 2013 IEEE International Conference on Computer Vision Workshops, pp. 354–361 (2013). https://doi.org/10.1109/ICCVW.2013.54

  39. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization (2014). Preprint at arXiv:1412.6980

Download references

Funding

No funding was received for conducting this study.

Author information

Author notes

  1. Jie Xiu and Xiujie Qu have authors contributed equally to this work.

Authors and Affiliations

  1. School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, 100081, China

    Jie Xiu, Xiujie Qu & Haowei Yu

Authors
  1. Jie Xiu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Xiujie Qu
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Haowei Yu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

All authors contributed to the study conception and design. Material preparation, data collection and analysis were performed by XJ, QX and YH. The first draft of the manuscript was written by XJ and all authors commented on previous versions of the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Jie Xiu.

Ethics declarations

Conflict of interest

The authors have no relevant financial or non-financial interests to disclose.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Xiu, J., Qu, X. & Yu, H. Double discriminative face super-resolution network with facial landmark heatmaps. Vis Comput 39, 5883–5895 (2023). https://doi.org/10.1007/s00371-022-02701-0

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00371-022-02701-0

Keywords

Search

Navigation