research-article

Deep Illumination-Enhanced Face Super-Resolution Network for Low-Light Images

Authors:

Xiaoyan KuiAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 18, Issue 3

Article No.: 87, Pages 1 - 19

https://doi.org/10.1145/3495258

Published: 04 March 2022 Publication History

Abstract

Face images are typically a key component in the fields of security and criminal investigation. However, due to lighting and shooting angles, faces taken under low-light conditions are often difficult to recognize. Face super-resolution (FSR) technology can restore high-resolution faces based on low-resolution inputs. However, existing face super-resolution methods typically rely on prior knowledge of inaccurate faces estimated from low-resolution images. Faces restored by low-light inputs may suffer from problems such as low brightness and many missing details. In this article, we proposed an Illumination-Enhanced Face Super-Resolution (IEFSR) model that can progressively super-resolve low-light faces of 32 × 32 pixels by an upscaling factor of 8. While reconstructing the low-light low-resolution face into a clear and high-quality face, we introduce a coarse low-resolution (LR) restoration network to recover the LR face details hidden in the dark. In the generator, we use a series of style blocks with noise to make the generated faces appear to have a more realistic visual aesthetic. Additionally, we introduce spectrum normalization in the discriminator to improve training stability. Extensive experimental evaluations show that the proposed IEFSR yields visually and metrically more attractive results than existing state-of-the-art FSR methods.

References

[1]

Honghao Gao and Yudong Zhang. 2020. Introduction to the special issue on smart communications and networking for future video surveillance. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 58 (Jul. 2020), 1–2. DOI:

Digital Library

[2]

Wen-Kuang Kuo and Kuo-Wei Wu. 2011. Traffic prediction and QoS transmission of real-time live VBR videos in WLANs. ACM Transactions on Multimedia Computing, Communications, and Applications 7, 36 (Nov. 2011), 1–21. DOI:

Digital Library

[3]

Nengjun Zhu, Jian Cao, Kunwei Shen, Xiaosong Chen, and Siji Zhu. 2020. A decision support system with intelligent recommendation for multi-disciplinary medical treatment. ACM Transactions on Multimedia Computing, Communications, and Applications 16, 33 (Apr. 2020), 1–23. DOI:

Digital Library

[4]

Vicki Bruce and Andy Young. 1986. Understanding face recognition. British Journal of Psychology. 77, 3 (Aug. 1986), 305–327. DOI:

[5]

Jeffrey R. Bach, Charles Fuller, and Amarnath Gupta. 1996. Virage image search engine: an open framework for image management. In Proceedings of the SPIE. Storage and Retrieval for Still Image and Video Databases, Vol. IV, International Society for Optics and Photonics, 76–87. DOI:

[6]

Michal Irani and Shmuel Pele. 1991. Improving resolution by image registration. Graphical Models and Image Processing. 53, 3 (May. 1991), 231–239. DOI:

Digital Library

[7]

Kui. Jia and Shaogang Gong. 2008. Generalized face super-resolution. IEEE Transactions on Image Processing 17, 6 (Jun. 2008), 873–886. DOI:

Digital Library

[8]

Hua Huang, Huiting He, and Xin Fan. 2010. Super-resolution of human face image using canonical correlation analysis. Pattern Recognition 43, 7 (Feb. 2010), 2532--2543. DOI:

Digital Library

[9]

Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. 2016. Deep cascaded bi-network for face hallucination. In Proceedings of the European Conference on Computer Vision. 614–630. DOI:

[10]

Xin Yu, Basura Fernando, and Bernard Ghanem. 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European Conference on Computer Vision. 217–233. DOI:

[11]

Yu Chen, Ying Tai, and Xiaoming Liu. 2018. FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2492–2501. DOI:

[12]

Feifan Lv, Feng Lu, Jianhua Wu, and Chongsoon Lim. 2018. MBLLEN: Low-light image/video enhancement using CNNs. In Proceedings of the British Machine Vision Conference. 220–233.

[13]

Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. 2018. Spectral normalization for generative adversarial networks. In Proceedings of the International Conference on Learning Representation. 1–26.

[14]

Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 908–917. DOI:

[15]

Cheng-Han Lee, Kaipeng Zhang, and Hu-Cheng Lee. 2018. Attribute augmented convolutional neural network for face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 721–729. DOI:

[16]

Ian Goodfellow, Jean Pouget-Abadie, and Mehdi Mirza. 2014. Generative adversarial networks. Advances in Neural Information Processing Systems 27 (2014), 2672–2680.

Digital Library

[17]

Christian Ledig, Lucas Theis, and Ferenc Huszár. 2017. Photo-Realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681–4690. DOI:

[18]

Xintao Wang, Ke Yu, and Shixiang Wu. 2018. ESRGAN: Enhanced super-resolution generative adversarial networks. In Proceedings of the European Conference on Computer Vision. 1–16. DOI:

[19]

Adrian Bulat and Georgios Tzimiropoulos. 2018. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low-resolution faces in arbitrary poses with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109–117. DOI:

[20]

Cheng Ma, Zhenyu Jiang, and Yongming Rao. 2020. Deep face super-resolution with iterative collaboration between attentive recovery and landmark estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5569–5578. DOI:

[21]

Ting-Chun Wang, Ming-Yu Liu, Jun-Yan Zhu, Andrew Tao, Jan Kautz, and Bryan Catanzaro. 2017. High-resolution image synthesis and semantic manipulation with conditional GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8798–8807.

[22]

Tero Karras, Timo, Aila, Samuli Laine, and Jaakko Lehtinen. 2018. Progressive growing of GANs for improved quality, stability, and variation. In Proceedings of the International Conference on Learning Representations. 1–26.

[23]

Tero Karras, Samuli Laine, and Timo Aila. 2019. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4401–4410. DOI:

[24]

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. 2020. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8110–8119.

[25]

Chaofeng Chen, Dihong Gong, Hao Wang, Zhifeng Li, and Kwan-Yee K. Wong. 2021. Learning spatial attention for face super-resolution. IEEE Transactions on Image Processing 30 (2021), 1219–1231. DOI:

Digital Library

[26]

Etta D. Pisano and Shuquan Zong. 1998. Contrast limited adaptive histogram equalization image processing to improve the detection of simulated spiculations in dense mammograms. Journal of Digital Imaging 11, 4 (Nov. 1998), 193–200.

[27]

Xuan Dong, Guan Wang, and Yi Pang. 2011. Fast efficient algorithm for enhancement of low lighting video. In Proceedings of the IEEE International Conference on Multimedia and Expo. 1–6. DOI:

Digital Library

[28]

Edwin H. Land. 1977. The retinex theory of color vision. Scientific American. 237, 6 (Dec. 1977), 108–128.

[29]

Daniel J. Jobson, Zia-ur Rahman, and Glenn A. Woodell. 1997. Properties and performance of a center/surround retinex. IEEE Transactions on Image Processing 6, 3 (Mar. 1997), 451–462. DOI:

Digital Library

[30]

Daniel J. Jobson, Zia-ur Rahman, and Glenn A. Woodell. 1997. A multiscale retinex for bridging the gap between color images and the human observation of scenes. IEEE Transactions on Image Processing. 6, 7 (Jul. 1997), 965–976. DOI:

Digital Library

[31]

Xiaojie Guo, Yu Li, and Haibin Ling. 2016. LIME: Low-light image enhancement via illumination map estimation. IEEE Transactions on Image Processing 26, 2 (Feb. 2016), 982–993. DOI:

Digital Library

[32]

Kin Gwn Lore, Adedotun Akintayo, and Soumik Sarkar. 2017. LLNet: A deep autoencoder approach to natural low-light image enhancement. Pattern Recognition 61 (2017), 650–662. DOI:

Digital Library

[33]

Chen Chen, Qifeng Chen, and Jia Xu. 2018. Learning to see in the dark. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3291–3300. DOI:

[34]

Li Tao and Chuang Zhu. 2017. LLCNN: A convolutional neural network for low-light image enhancement. In Proceedings of theIEEE Visual Communications and Image Processing. 1–4. DOI:

[35]

Chen Wei, Wenjing Wang, and Wenhan Yang. 2018. Deep retinex decomposition for low-light enhancement. In Proceedings of the British Machine Vision Conference. 127–136.

[36]

Ruixing Wang, Qing Zhang, and Chi-Wing Fu. 2019. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 6849–6857. DOI:

[37]

Martin Heusel, Hubert Ramsauer, and Thomas Unterthiner. 2017. GANs trained by a two time-scale update rule converge to a local Nash equilibrium. In Proceedings of the 31st International Conference on Neural Information Processing Systems. 6626–6637.

[38]

Tero Karen Simonyan and Andrew Zisserman. 2014. Very deep convolutional networks for large-scale image recognition. In Proceedings of the 3rd International Conference on Learning Representations. 1–14.

[39]

Jongyeon Lee. 2020. Face super-resolution with styled feature channel attention. Hongik University Graduation Project. Retrieved Sept. 27, 2020, from https://github.com/tinnunculus/Styled-Attention-Face-Super-Resolution.

[40]

Yulun Zhang, Kunpeng Li, and Kai Li. 2018. Image super-resolution using very deep residual channel attention networks. In Proceedings of the European Conference on Computer Vision. 286–301. DOI:

[41]

Han Zhang, Ian Goodfellow, and Dimitris Metaxas. 2019. Self-attention generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 7354–7363.

[42]

Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2D & 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the IEEE International Conference on Computer Vision. 1021–1030. DOI:

[43]

Q Huynh-Thu and Mohammed Ghanbari. 2008. Scope of validity of PSNR in image/video quality assessment. Electronics Letters 44, 13 (Feb. 2008), 800–801. DOI:

[44]

Zhou Wang, Alan Conrad Bovik, and Hamid Rahim Sheikh. 2004. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing 13, 4 (Apr. 2004), 600–612. DOI:

Digital Library

[45]

Muhammad Haris, Greg Shakhnarovich, and Norimichi Ukita. 2018. Deep back-projection networks for super-resolution. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1664–1673. DOI:

[46]

Xintao Wang, Yu Li, Honglun Zhang, and Ying Shan. 2021. Towards real-world blind face restoration with generative facial prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 9164--9174. DOI:

[47]

Weili Guan, Zhaozheng Chen, Fuli Feng, Weifeng Liu, and Liqiang Nie. 2021. Urban perception: Sensing cities via a deep interactive multi-task learning framework. ACM Transactions on Multimedia Computing, Communications, and Applications 17, 1 (Jan. 2021), Article 13, 20 pages. DOI:

Digital Library

[48]

Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. 2017. VoxCeleb: A large-scale speaker identification dataset. In Proceedings of the Conference of the International Speech Communication Association. 1–6.

[49]

Nima Khademi Kalantari and Ravi Ramamoorthi. 2017. Deep high dynamic range imaging of dynamic scenes. ACM Transactions on Graphics 36, 4 (Jul. 2017), Article 144, 12 pages. DOI:

Digital Library

Cited By

Pan JLiu XBai YZhai DJiang JZhao D(2024)Illumination-Aware Low-Light Image Enhancement with Transformer and Auto-Knee CurveACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366465320:8(1-23)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3664653
Wang YLu TYao YZhang YXiong Z(2024)Learning to Hallucinate Face in the DarkIEEE Transactions on Multimedia10.1109/TMM.2023.329480826(2314-2326)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3294808
Jeon JPark JEom I(2024)Low-light image enhancement using gamma correction prior in mixed color spacesPattern Recognition10.1016/j.patcog.2023.110001146:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110001
Show More Cited By

Index Terms

Deep Illumination-Enhanced Face Super-Resolution Network for Low-Light Images
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
      1. Image processing

Recommendations

Eigentransformation-based face super-resolution in the wavelet domain

In this paper, we propose a wavelet-based eigentransformation method for human face hallucination. Our algorithm uses the wavelet transform to decompose interpolated low-resolution (LR) images in the wavelet domain to obtain high-frequency information ...
Double discriminative face super-resolution network with facial landmark heatmaps
Abstract
At present, most of face super-resolution (SR) networks cannot balance the visual quality and the pixel accuracy. The networks with high objective index values often reconstruct too smooth images, while the networks which can restore texture ...
Learning the Relationship Between High and Low Resolution Images in Kernel Space for Face Super Resolution
ICPR '10: Proceedings of the 2010 20th International Conference on Pattern Recognition

This paper proposes a new nonlinear face super resolution algorithm to address an important issue in face recognition from surveillance video namely, recognition of low resolution face image with nonlinear variations. The proposed method learns the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 18, Issue 3

August 2022

478 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3505208

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2022

Accepted: 01 November 2021

Revised: 01 October 2021

Received: 01 May 2021

Published in TOMM Volume 18, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Refereed

Funding Sources

National Natural Science Foundation of China
Hunan Provincial Science and Technology Plan Project
National Science Foundation of Hunan Province
National Social Science Fund of China
Postgraduate Scientific Research Innovation Project of Hunan Province
Fundamental Research Funds for the Central Universities of Central South University
Changsha Municipal Natural Science Foundation

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
948
Total Downloads

Downloads (Last 12 months)254
Downloads (Last 6 weeks)9

Reflects downloads up to 27 Jul 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pan JLiu XBai YZhai DJiang JZhao D(2024)Illumination-Aware Low-Light Image Enhancement with Transformer and Auto-Knee CurveACM Transactions on Multimedia Computing, Communications, and Applications10.1145/366465320:8(1-23)Online publication date: 29-Jun-2024
https://dl.acm.org/doi/10.1145/3664653
Wang YLu TYao YZhang YXiong Z(2024)Learning to Hallucinate Face in the DarkIEEE Transactions on Multimedia10.1109/TMM.2023.329480826(2314-2326)Online publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1109/TMM.2023.3294808
Jeon JPark JEom I(2024)Low-light image enhancement using gamma correction prior in mixed color spacesPattern Recognition10.1016/j.patcog.2023.110001146:COnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.patcog.2023.110001
Ye JLiu SQiu CZhang Z(2024)Learning multi-granularity semantic interactive representation for joint low-light image enhancement and super-resolutionInformation Fusion10.1016/j.inffus.2024.102467110(102467)Online publication date: Oct-2024
https://doi.org/10.1016/j.inffus.2024.102467
Rajput SRai DKumar B(2024)OEINR-RFHExpert Systems with Applications: An International Journal10.1016/j.eswa.2023.121553237:PCOnline publication date: 1-Feb-2024
https://dl.acm.org/doi/10.1016/j.eswa.2023.121553
Guo KZhang ZGuo HRen SWang LZhou XLiu C(2023)Video Super-Resolution Based on Inter-Frame Information Utilization for Intelligent TransportationIEEE Transactions on Intelligent Transportation Systems10.1109/TITS.2023.323770824:11(13409-13421)Online publication date: 23-Jan-2023
https://dl.acm.org/doi/10.1109/TITS.2023.3237708
Rai DRajput S(2023)Low-Light Robust Face Image Super-Resolution via Neuro-Fuzzy Inferencing-Based Locality Constrained RepresentationIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2023.328053772(1-11)Online publication date: 2023
https://doi.org/10.1109/TIM.2023.3280537
Abbas FMalah MBabahenini M(2023)Approximating global illumination with ambient occlusion and environment light via generative adversarial networksPattern Recognition Letters10.1016/j.patrec.2022.12.007166:C(209-217)Online publication date: 1-Feb-2023
https://dl.acm.org/doi/10.1016/j.patrec.2022.12.007
Wang HChi JWu CYu XWu H(2023)Progressive local-to-global vision transformer for occluded face hallucinationMultimedia Tools and Applications10.1007/s11042-023-15028-283:3(8219-8240)Online publication date: 15-Jun-2023
https://dl.acm.org/doi/10.1007/s11042-023-15028-2
Tomar AArya KRajput S(2023)Noise robust face super-resolution via learning of spatial attentive featuresMultimedia Tools and Applications10.1007/s11042-023-14472-482:16(25449-25465)Online publication date: 21-Feb-2023
https://dl.acm.org/doi/10.1007/s11042-023-14472-4
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Full Text

View this article in Full Text.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View full text|Download PDF

View Issue’s Table of Contents