research-article

A Deep Learning Approach for Face Hallucination Guided by Facial Boundary Responses

Authors:

Jun YuAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 16, Issue 1

Article No.: 17, Pages 1 - 23

https://doi.org/10.1145/3377874

Published: 04 March 2020 Publication History

Abstract

Face hallucination is a domain-specific super-resolution (SR) problem of learning a mapping between a low-resolution (LR) face image and its corresponding high-resolution (HR) image. Tremendous progress on deep learning has shown exciting potential for a variety of face hallucination tasks. However, most deep-learning–based methods are limited to handle facial appearance information without paying attention to facial structure priors. In this article, we propose an open source¹ Boundary-aware Dual-branch Network (BDN) for face hallucination, which simultaneously extracts face features and estimates facial boundary responses from LR inputs, ultimately fusing them to reconstruct HR results. Specifically, we first upsample LR face images to HR feature maps, and then feed the upsampled HR features into a memory unit and an attention unit synchronously to obtain the refined features and predict facial boundary responses. Next, they are fed into a feature map fusion unit to combine facial appearance and structure information by a spatial attention mechanism. Moreover, we employ a series of stacked units to boost performance before recovering HR face images. Finally, a discriminative network is developed to improve visual quality by introducing adversarial learning strategy. Extensive experiments show that the proposed approach achieves superior face hallucination results against the state-of-the-art ones.

References

[1]

Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein generative adversarial networks. In Proceedings of the International Conference on Machine Learning. 214--223.

[2]

Simon Baker and Takeo Kanade. 2000. Hallucinating faces. In Proceedings of the IEEE International Conference on Automatic Face and Gesture Recognition (FG’00). 83--88.

[3]

Simon Baker and Takeo Kanade. 2002. Limits on super-resolution and how to break them. IEEE Trans. Pattern Anal. Mach. Intell. 9 (2002), 1167--1183.

Digital Library

[4]

Adrian Bulat and Georgios Tzimiropoulos. 2017. How far are we from solving the 2D 8 3D face alignment problem? (and a dataset of 230,000 3D facial landmarks). In Proceedings of the International Conference on Computer Vision.

[5]

Adrian Bulat and Georgios Tzimiropoulos. 2018. Super-FAN: Integrated facial landmark localization and super-resolution of real-world low resolution faces in arbitrary poses with GANs. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 109--117.

[6]

Qingxing Cao, Liang Lin, Yukai Shi, Xiaodan Liang, and Guanbin Li. 2017. Attention-aware face hallucination via deep reinforcement learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 690--698.

[7]

Yu Chen, Ying Tai, Xiaoming Liu, Chunhua Shen, and Jian Yang. 2018. FSRNet: End-to-end learning face super-resolution with facial priors. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2492--2501.

[8]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789--8797.

[9]

Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A. Bharath. 2018. Generative adversarial networks: An overview. IEEE Sig. Proc. Mag. 35, 1 (2018), 53--65.

[10]

Peter Dayan, Laurence F. Abbott, and L. Abbott. 2001. Theoretical neuroscience: Computational and mathematical modeling of neural systems. Journal of Cognitive Neuroscience 15, 1 (2003), 154--155.

[11]

Berk Dogan, Shuhang Gu, and Radu Timofte. 2019. Exemplar guided face image super-resolution without facial landmarks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) Workshops.

[12]

Chao Dong, Chen Change Loy, Kaiming He, and Xiaoou Tang. 2016. Image super-resolution using deep convolutional networks. IEEE Trans. Pattern Anal. 8 Mach. Intell. 38, 2 (2016), 295--307.

Digital Library

[13]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Proceedings of the International Conference on Advances in Neural Information Processing Systems. 2672--2680.

[14]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 770--778.

[15]

Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q. Weinberger. 2017. Densely connected convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4700--4708.

[16]

Gary B. Huang, Marwan Mattar, Tamara Berg, and Eric Learned-Miller. 2008. Labeled faces in the wild: A database for studying face recognition in unconstrained environments. In Proceedings of the Workshop on Faces in “Real-Life” Images: Detection, Alignment, and Recognition.

[17]

Huaibo Huang, Ran He, Zhenan Sun, and Tieniu Tan. 2017. Wavelet-SRNet: A wavelet-based CNN for multi-scale face super resolution. In Proceedings of the IEEE International Conference on Computer Vision. 1689--1697.

[18]

Andrey Ignatov, Nikolay Kobyshev, Radu Timofte, Kenneth Vanhoey, and Luc Van Gool. 2018. WESPE: Weakly supervised photo enhancer for digital cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops. 691--700.

[19]

Satoshi Iizuka, Edgar Simo-Serra, and Hiroshi Ishikawa. 2017. Globally and locally consistent image completion. ACM Trans. Graph. 36, 4 (2017), 107.

Digital Library

[20]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A. Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1125--1134.

[21]

Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In Proceedings of the European Conference on Computer Vision. Springer, 694--711.

[22]

Jiwon Kim, Jung Kwon Lee, and Kyoung Mu Lee. 2016. Accurate image super-resolution using very deep convolutional networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1646--1654.

[23]

Soheil Kolouri and Gustavo K. Rohde. 2015. Transport-based single frame super resolution of very low resolution face images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4876--4884.

[24]

Lingchao Kong and Rui Dai. 2018. Efficient video encoding for automatic video analysis in distributed wireless surveillance systems. ACM Trans. Multim. Comput. Commun. Applic. 14, 3 (2018), 72.

[25]

Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-realistic single image super-resolution using a generative adversarial network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4681--4690.

[26]

Mengyan Li, Yuechuan Sun, Zhaoyu Zhang, and Jun Yu. 2018. A coarse-to-fine face hallucination method by exploiting facial prior knowledge. In Proceedings of the 25th IEEE International Conference on Image Processing (ICIP’18). IEEE, 61--65.

[27]

Xianguo Li, Yemei Sun, Yanli Yang, and Changyun Miao. 2019. Symmetrical residual connections for single image super-resolution. ACM Trans. Multim. Comput. Commun. Applic. 15, 1 (2019), 19.

[28]

Yongchao Li, Cheng Cai, Guoping Qiu, and Kin-Man Lam. 2014. Face hallucination based on sparse local-pixel structure. Pattern Recog. 47, 3 (2014), 1261--1270.

Digital Library

[29]

Ming Liang and Xiaolin Hu. 2015. Recurrent convolutional neural network for object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3367--3375.

[30]

Ce Liu, Heung-Yeung Shum, and Chang-Shui Zhang. 2001. A two-step approach to hallucinating faces: Global parametric model and local nonparametric model. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR’01). Citeseer, 192--198.

[31]

Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2015. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3730--3738.

Digital Library

[32]

Yongyi Lu, Yu-Wing Tai, and Chi-Keung Tang. 2018. Attribute-guided face generation using conditional CycleGAN. In Proceedings of the European Conference on Computer Vision (ECCV’18). 282--297.

[33]

Xiang Ma, Junping Zhang, and Chun Qi. 2010. Hallucinating face by position-patch. Pattern Recog. 43, 6 (2010), 2224--2236.

Digital Library

[34]

Iacopo Masi, Feng-Ju Chang, Jongmoo Choi, Shai Harel, Jungyeon Kim, Kang Geon Kim, Jatuporn Leksut, Stephen Rawls, Yue Wu, Tal Hassner, et al. 2019. Learning pose-aware models for pose-invariant face recognition in the wild. IEEE Trans. Pattern Anal. 8 Mach. Intell. 41, 2 (2019), 379--393.

Digital Library

[35]

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, et al. 2015. Deep face recognition. In Proceedings of the British Machine Vision Conference, Vol. 1. 6.

[36]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 815--823.

[37]

Wenzhe Shi, Jose Caballero, Ferenc Huszár, Johannes Totz, Andrew P. Aitken, Rob Bishop, Daniel Rueckert, and Zehan Wang. 2016. Real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1874--1883.

[38]

Ying Tai, Jian Yang, Xiaoming Liu, and Chunyan Xu. 2017. Memnet: A persistent memory network for image restoration. In Proceedings of the IEEE International Conference on Computer Vision. 4539--4547.

[39]

Zhiqiang Tang, Xi Peng, Shijie Geng, Lingfei Wu, Shaoting Zhang, and Dimitris Metaxas. 2018. Quantized densely connected u-nets for efficient landmark localization. In Proceedings of the European Conference on Computer Vision (ECCV’18). 339--354.

[40]

Nannan Wang, Dacheng Tao, Xinbo Gao, Xuelong Li, and Jie Li. 2014. A comprehensive survey to face hallucination. Int. J. Comput. Vis. 106, 1 (2014), 9--30.

Digital Library

[41]

Xiaogang Wang and Xiaoou Tang. 2005. Hallucinating face by eigentransformation. IEEE Trans. Syst. Man. Cyber. Part C (Applic. Rev.) 35, 3 (2005), 425--434.

Digital Library

[42]

Wayne Wu, Chen Qian, Shuo Yang, Quan Wang, Yici Cai, and Qiang Zhou. 2018. Look at boundary: A boundary-aware face alignment algorithm. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2129--2138.

[43]

Taihong Xiao, Jiapeng Hong, and Jinwen Ma. 2018. ELEGANT: Exchanging latent encodings with GAN for transferring multiple face attributes. In Proceedings of the European Conference on Computer Vision (ECCV’18). 168--184.

[44]

Chih-Yuan Yang, Sifei Liu, and Ming-Hsuan Yang. 2013. Structured face hallucination. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1099--1106.

Digital Library

[45]

Raymond A. Yeh, Chen Chen, Teck Yian Lim, Alexander G. Schwing, Mark Hasegawa-Johnson, and Minh N. Do. 2017. Semantic image inpainting with deep generative models. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 5485--5493.

[46]

Xin Yu, Basura Fernando, Bernard Ghanem, Fatih Porikli, and Richard Hartley. 2018. Face super-resolution guided by facial component heatmaps. In Proceedings of the European Conference on Computer Vision (ECCV’18). 217--233.

[47]

Xin Yu, Basura Fernando, Richard Hartley, and Fatih Porikli. 2018. Super-resolving very low-resolution face images with supplementary attributes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 908--917.

[48]

Xin Yu and Fatih Porikli. 2016. Ultra-resolving face images by discriminative generative networks. In Proceedings of the European Conference on Computer Vision. Springer, 318--333.

[49]

Xin Yu and Fatih Porikli. 2017. Face hallucination with tiny unaligned images by transformative discriminative neural networks. In Proceedings of the 31st AAAI Conference on Artificial Intelligence.

[50]

Xin Yu and Fatih Porikli. 2017. Hallucinating very low-resolution unaligned and noisy face images by transformative discriminative autoencoders. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3760--3768.

[51]

Xin Yu and Fatih Porikli. 2018. Imagining the unimaginable faces by deconvolutional networks. IEEE Trans. Image Proc. 27, 6 (2018), 2747--2761.

Digital Library

[52]

Xin Yu, Fatih Porikli, Basura Fernando, and Richard Hartley. 2020. Hallucinating unaligned face images by multiscale transformative discriminative networks. Int. J. Comput. Vis. 128, 2 (2020), 500--526.

[53]

Kaipeng Zhang, Zhanpeng Zhang, Chia-Wen Cheng, Winston H. Hsu, Yu Qiao, Wei Liu, and Tong Zhang. 2018. Super-identity convolutional neural network for face hallucination. In Proceedings of the European Conference on Computer Vision (ECCV’18). 183--198.

[54]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Sig. Proc. Lett. 23, 10 (2016), 1499--1503.

[55]

Erjin Zhou, Haoqiang Fan, Zhimin Cao, Yuning Jiang, and Qi Yin. 2015. Learning face hallucination in the wild. In Proceedings of the 29th AAAI Conference on Artificial Intelligence.

[56]

Joey Tianyi Zhou, Jiawei Du, Hongyuan Zhu, Xi Peng, Yong Liu, and Rick Siow Mong Goh. 2019. AnomalyNet: An anomaly detection network for video surveillance. IEEE Trans. Inf. Forens. Secur. 14, 10 (2019), 2537--2550.

[57]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision. 2223--2232.

[58]

Shizhan Zhu, Sifei Liu, Chen Change Loy, and Xiaoou Tang. 2016. Deep cascaded bi-network for face hallucination. In Proceedings of the European Conference on Computer Vision. Springer, 614--630.

Cited By

Guo KChen LZhu XKui XZhang JShi H(2023)Double-layer Search and Adaptive Pooling Fusion for Reference-based Image Super-resolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3604937Online publication date: 21-Jun-2023
https://doi.org/10.1145/3604937
Kordon FStiglmayr MMaier AMartín Vicario CPertlwieser TKunze H(2023)A principled representation of elongated structures using heatmapsScientific Reports10.1038/s41598-023-41221-213:1Online publication date: 14-Sep-2023
https://doi.org/10.1038/s41598-023-41221-2
Gao GYu YYang JQi GYang M(2022)Hierarchical Deep CNN Feature Set-Based Representation Learning for Robust Cross-Resolution Face RecognitionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2020.304217832:5(2550-2560)Online publication date: May-2022
https://doi.org/10.1109/TCSVT.2020.3042178

Index Terms

A Deep Learning Approach for Face Hallucination Guided by Facial Boundary Responses
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision

Recommendations

A novel correspondence-based face-hallucination method

This paper addresses the problem of estimating high-resolution (HR) facial images from a single low-resolution (LR) input. We assume that the input LR and estimated HR images are under the same view-point and illumination condition, i.e. the setting of ...
An Example-Based Face Hallucination Method for Single-Frame, Low-Resolution Facial Images

This paper proposes a face hallucination method for the reconstruction of high-resolution facial images from single-frame, low-resolution facial images. The proposed method has been derived from example-based hallucination methods and morphable face ...
A Comprehensive Survey to Face Hallucination

This paper comprehensively surveys the development of face hallucination (FH), including both face super-resolution and face sketch-photo synthesis techniques. Indeed, these two techniques share the same objective of inferring a target face image (e.g. ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 16, Issue 1

February 2020

363 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3384216

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 March 2020

Accepted: 01 December 2019

Revised: 01 November 2019

Received: 01 April 2019

Published in TOMM Volume 16, Issue 1

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China
USTC Research Funds of the Double First-Class Initiative

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

3
Total Citations
View Citations
304
Total Downloads

Downloads (Last 12 months)21
Downloads (Last 6 weeks)6

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Guo KChen LZhu XKui XZhang JShi H(2023)Double-layer Search and Adaptive Pooling Fusion for Reference-based Image Super-resolutionACM Transactions on Multimedia Computing, Communications, and Applications10.1145/3604937Online publication date: 21-Jun-2023
https://doi.org/10.1145/3604937
Kordon FStiglmayr MMaier AMartín Vicario CPertlwieser TKunze H(2023)A principled representation of elongated structures using heatmapsScientific Reports10.1038/s41598-023-41221-213:1Online publication date: 14-Sep-2023
https://doi.org/10.1038/s41598-023-41221-2
Gao GYu YYang JQi GYang M(2022)Hierarchical Deep CNN Feature Set-Based Representation Learning for Robust Cross-Resolution Face RecognitionIEEE Transactions on Circuits and Systems for Video Technology10.1109/TCSVT.2020.304217832:5(2550-2560)Online publication date: May-2022
https://doi.org/10.1109/TCSVT.2020.3042178

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Media

Figures

Other

Tables

View Issue’s Table of Contents