Article

Free access

Dual-agent GANs for photorealistic and identity preserving profile face synthesis

Authors:

Karlekar Jayashree,

Sugiri Pranata,

Jiashi FengAuthors Info & Claims

NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems

Pages 65 - 75

Published: 04 December 2017 Publication History

PDF eReader Publisher Site

Abstract

Synthesizing realistic profile faces is promising for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by populating samples with extreme poses and avoiding tedious annotations. However, learning from synthetic faces may not achieve the desired performance due to the discrepancy between distributions of the synthetic and real face images. To narrow this gap, we propose a Dual-Agent Generative Adversarial Network (DA-GAN) model, which can improve the realism of a face simulator's output using unlabeled real faces, while preserving the identity information during the realism refinement. The dual agents are specifically designed for distinguishing real v.s. fake and identities simultaneously. In particular, we employ an off-the-shelf 3D face model as a simulator to generate profile face images with varying poses. DA-GAN leverages a fully convolutional network as the generator to generate high-resolution images and an auto-encoder as the discriminator with the dual agents. Besides the novel architecture, we make several key modifications to the standard GAN to preserve pose and texture, preserve identity and stabilize training process: (i) a pose perception loss; (ii) an identity perception loss; (iii) an adversarial loss with a boundary equilibrium regularization term. Experimental results show that DA-GAN not only presents compelling perceptual results but also significantly outperforms state-of-the-arts on the large-scale and challenging NISTIJB-A unconstrained face recognition benchmark. In addition, the proposed DA-GAN is also promising as a new approach for solving generic transfer learning problems more effectively. DA-GAN is the foundation of our submissions to NIST IJB-A 2017 face recognition competitions, where we won the 1^st places on the tracks of verification and identification.

References

[1]

W. AbdAlmageed, Y. Wu, S. Rawls, S. Harel, T. Hassner, I. Masi, J. Choi, J. Lekust, J. Kim, P. Natarajan, et al. Face recognition using deep multi-pose representations. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016.

[2]

D. Berthelot, T. Schumm, and L. Metz. Began: Boundary equilibrium generative adversarial networks. arXiv preprint arXiv:1703.10717, 2017.

[3]

J.-C. Chen, V. M. Patel, and R. Chellappa. Unconstrained face verification using deep cnn features. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016.

[4]

J.-C. Chen, R. Ranjan, A. Kumar, C.-H. Chen, V. M. Patel, and R. Chellappa. An end-to-end system for unconstrained face verification with deep convolutional neural networks. In Proceedings of the IEEE International Conference on Computer Vision Workshops (CVPRW), pages 118-126, 2015.

Digital Library

[5]

X. Chen, Y. Duan, R. Houthooft, J. Schulman, I. Sutskever, and P. Abbeel. Infogan: Interpretable representation learning by information maximizing generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), pages 2172-2180, 2016.

[6]

F. Chollet. keras. https://github.com/fchollet/keras, 2015.

[7]

A. R. Chowdhury, T.-Y. Lin, S. Maji, and E. Learned-Miller. One-to-many face recognition with bilinear cnns. In Proceedings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pages 1-9, 2016.

[8]

N. Crosswhite, J. Byrne, O. M. Parkhi, C. Stauffer, Q. Cao, and A. Zisserman. Template adaptation for face verification and identification. arXiv preprint arXiv:1603.03958, 2016.

[9]

K. Gong, X. Liang, X. Shen, and L. Lin. Look into person: Self-supervised structure-sensitive learning and a new benchmark for human parsing. arXiv preprint arXiv:1703.05446, 2017.

[10]

I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial nets. In Proceedings of the Advances in Neural Information Processing Systems (NIPS), pages 2672-2680, 2014.

Digital Library

[11]

T. Hassner, I. Masi, J. Kim, J. Choi, S. Harel, P. Natarajan, and G. Medioni. Pooling faces: template based face recognition with pooled face images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 59-67, 2016.

[12]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.

[13]

R. Huang, S. Zhang, T. Li, and R. He. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. arXiv preprint arXiv:1704.04086, 2017.

[14]

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.

[15]

B. F. Klare, B. Klein, E. Taborsky, A. Blanton, J. Cheney, K. Allen, P. Grother, A. Mah, M. Burge, and A. K. Jain. Pushing the frontiers of unconstrained face detection and recognition: Iarpa janus benchmark a. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1931-1939, 2015.

[16]

J. Li, J. Zhao, F. Zhao, H. Liu, J. Li, S. Shen, J. Feng, and T. Sim. Robust face recognition with deep multi-view representation learning. In Proceedings of the ACM Conference on Multimedia (ACM MM), pages 1068-1072, 2016.

Digital Library

[17]

J. Long, E. Shelhamer, and T. Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3431-3440, 2015.

[18]

L. v. d. Maaten and G. Hinton. Visualizing data using t-sne. Journal of Machine Learning Research (JMLR), 9(Nov):2579-2605, 2008.

[19]

I. Masi, S. Rawls, G. Medioni, and P. Natarajan. Pose-aware face recognition in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4838-4846, 2016.

[20]

I. Masi, A. T. Tran, J. T. Leksut, T. Hassner, and G. Medioni. Do we really need to collect millions of faces for effective face recognition? arXiv preprint arXiv:1603.07057, 2016.

[21]

M. Mirza and S. Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784, 2014.

[22]

A. Odena, C. Olah, and J. Shlens. Conditional image synthesis with auxiliary classifier gans. arXiv preprint arXiv:1610.09585, 2016.

[23]

O. M. Parkhi, A. Vedaldi, and A. Zisserman. Deep face recognition. In Proceedings of the British Machine Vision Conference (BMVC), page 6, 2015.

[24]

R. Ranjan, C. D. Castillo, and R. Chellappa. L2-constrained softmax loss for discriminative face verification. arXiv preprint arXiv:1703.09507, 2017.

[25]

R. Ranjan, S. Sankaranarayanan, C. D. Castillo, and R. Chellappa. An all-in-one convolutional neural network for face analysis. arXiv preprint arXiv:1611.00851, 2016.

[26]

D. J. Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. arXiv preprint arXiv:1401.4082, 2014.

[27]

S. Sankaranarayanan, A. Alavi, C. D. Castillo, and R. Chellappa. Triplet probabilistic embedding for face verification and clustering. In Proceedings of the IEEE Conference on Biometrics: Theory, Applications and Systems (BTAS), pages 1-8, 2016.

[28]

A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R. Webb. Learning from simulated and unsupervised images through adversarial training. arXiv preprint arXiv:1612.07828, 2016.

[29]

Y. Taigman, M. Yang, M. Ranzato, and L. Wolf. Deepface: Closing the gap to human-level performance in face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 1701-1708, 2014.

Digital Library

[30]

D. Wang, C. Otto, and A. K. Jain. Face search at scale: 80 million gallery. arXiv preprint arXiv:1507.07242, 2015.

[31]

S. Xiao, J. Feng, J. Xing, H. Lai, S. Yan, and A. Kassim. Robust facial landmark detection via recurrent attentive-refinement networks. In Proceedings of the European Conference on Computer Vision (ECCV), pages 57-72, 2016.

[32]

S. Xiao, L. Liu, X. Nie, J. Feng, A. A. Kassim, and S. Yan. A live face swapper. In Proceedings of the ACM Conference on Multimedia (ACM MM), pages 691-692, 2016.

Digital Library

[33]

S. Xie, R. Girshick, P. Dollär, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. arXiv preprint arXiv:1611.05431, 2016.

[34]

J. Yang, P. Ren, D. Chen, F. Wen, H. Li, and G. Hua. Neural aggregation network for video face recognition. arXiv preprint arXiv:1603.05474, 2016.

[35]

X. Zhu, J. Yan, D. Yi, Z. Lei, and S. Z. Li. Discriminative 3d morphable model fitting. In Proceedings of the IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), volume 1, pages 1-8, 2015.

Cited By

Kammoun ASlama RTabia HOuni TAbid M(2022)Generative Adversarial Networks for Face Generation: A SurveyACM Computing Surveys10.1145/352785055:5(1-37)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3527850
Sun JLi QWang WZhao JSun ZShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Multi-caption Text-to-Face Synthesis: Dataset and AlgorithmProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475391(2290-2298)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475391
Chen YYang ZAbbou RLopes PZhao BZheng HKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)User Authentication via Electrical Muscle StimulationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445441(1-15)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445441
Show More Cited By

Recommendations

Deep Learning Identity-Preserving Face Space
ICCV '13: Proceedings of the 2013 IEEE International Conference on Computer Vision

Face recognition with large pose and illumination variations is a challenging problem in computer vision. This paper addresses this challenge by proposing a new learning based face representation: the face identity-preserving (FIP) features. Unlike ...
U-Net Conditional GANs for Photo-Realistic and Identity-Preserving Facial Expression Synthesis
Special Issue on Face Analysis for Applications and Special Issue on Affective Computing for Large-Scale Heterogeneous Multimedia Data

Facial expression synthesis (FES) is a challenging task since the expression changes are highly non-linear and depend on the facial appearance. Person identity should also be well preserved in the synthesized face. In this article, we present a novel U-...
3D-Aided Dual-Agent GANs for Unconstrained Face Recognition
Synthesizing realistic profile faces is beneficial for more efficiently training deep pose-invariant models for large-scale unconstrained face recognition, by augmenting the number of samples with extreme poses and avoiding costly annotation work. However,...

Comments

Information & Contributors

Information

Published In

cover image Guide Proceedings

NIPS'17: Proceedings of the 31st International Conference on Neural Information Processing Systems

December 2017

7104 pages

ISBN:9781510860964

Publisher

Curran Associates Inc.

Red Hook, NY, United States

Publication History

Published: 04 December 2017

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
106
Total Downloads

Downloads (Last 12 months)68
Downloads (Last 6 weeks)18

Reflects downloads up to 04 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Kammoun ASlama RTabia HOuni TAbid M(2022)Generative Adversarial Networks for Face Generation: A SurveyACM Computing Surveys10.1145/352785055:5(1-37)Online publication date: 3-Dec-2022
https://dl.acm.org/doi/10.1145/3527850
Sun JLi QWang WZhao JSun ZShen HZhuang YSmith JYang YCesar PMetze FPrabhakaran B(2021)Multi-caption Text-to-Face Synthesis: Dataset and AlgorithmProceedings of the 29th ACM International Conference on Multimedia10.1145/3474085.3475391(2290-2298)Online publication date: 17-Oct-2021
https://dl.acm.org/doi/10.1145/3474085.3475391
Chen YYang ZAbbou RLopes PZhao BZheng HKitamura YQuigley AIsbister KIgarashi TBjørn PDrucker S(2021)User Authentication via Electrical Muscle StimulationProceedings of the 2021 CHI Conference on Human Factors in Computing Systems10.1145/3411764.3445441(1-15)Online publication date: 6-May-2021
https://dl.acm.org/doi/10.1145/3411764.3445441
Zhao JLi JTu XZhao FXin YXing JLiu HYan SFeng J(2019)Multi-prototype networks for unconstrained set-based face recognitionProceedings of the 28th International Joint Conference on Artificial Intelligence10.5555/3367471.3367653(4397-4403)Online publication date: 10-Aug-2019
https://dl.acm.org/doi/10.5555/3367471.3367653
Zhang HZhou AXu DXu SZhang XMa H(2019)Learning to Recognize Unmodified Lights with Invisible FeaturesProceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies10.1145/33289383:2(1-23)Online publication date: 21-Jun-2019
https://dl.acm.org/doi/10.1145/3328938
Cao JHu YYu BHe RSun Z(2019)3D Aided Duet GANs for Multi-View Face Image SynthesisIEEE Transactions on Information Forensics and Security10.1109/TIFS.2019.289111614:8(2028-2042)Online publication date: 1-Aug-2019
https://dl.acm.org/doi/10.1109/TIFS.2019.2891116
Moniz JBeckham CRajotte SHonari SPal C(2018)Unsupervised depth estimation, 3D face rotation and replacementProceedings of the 32nd International Conference on Neural Information Processing Systems10.5555/3327546.3327642(9759-9769)Online publication date: 3-Dec-2018
https://dl.acm.org/doi/10.5555/3327546.3327642
Cao JHu YZhang HHe RSun Z(2018)Learning a high fidelity pose invariant model for high-resolution face frontalizationProceedings of the 32nd International Conference on Neural Information Processing Systems10.5555/3327144.3327210(2872-2882)Online publication date: 3-Dec-2018
https://dl.acm.org/doi/10.5555/3327144.3327210

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten