research-article

TC-GAN: Triangle Cycle-Consistent GANs for Face Frontalization with Facial Features Preserved

Authors:

Yi-Ping Phoebe Chen,

Yu-Gang JiangAuthors Info & Claims

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

Pages 220 - 228

https://doi.org/10.1145/3343031.3351031

Published: 15 October 2019 Publication History

Abstract

Face frontalization has always been an important field. Recently, with the introduction of generative adversarial networks (GANs), face frontalization has achieved remarkable success. A critical challenge during face frontalization is to ensure the features of the original profile image are retained. Even though some state-of-the-art methods can preserve identity features while rotating the face to the frontal view, they still have difficulty preserving facial expression features. Therefore, we propose the novel triangle cycle-consistent generative adversarial networks for the face frontalization task, termed TC-GAN. Our networks contain two generators and one discriminator. One of the generators generates the frontal contour, and the other generates the facial features. They work together to generate a photo-realistic frontal view of the face. We also introduce cycle-consistent loss to retain feature information effectively. To validate the advantages of TC-GAN, we apply it to the face frontalization task on two datasets. The experimental results demonstrate that our method can perform large-pose face frontalization while preserving the facial features (both identity and expression). To the best of our knowledge, TC-GAN outperforms the state-of-the-art methods in the preservation of facial identity and expression features during face frontalization.

References

[1]

Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017).

[2]

Volker Blanz, Thomas Vetter, et almbox. 1999. A morphable model for the synthesis of 3D faces. In Siggraph, Vol. 99. 187--194.

[3]

Dong Chen, Xudong Cao, Fang Wen, and Jian Sun. 2013. Blessing of dimensionality: High-dimensional feature and its efficient compression for face verification. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 3025--3032.

Digital Library

[4]

Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. 2018. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 8789--8797.

[5]

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.

[6]

Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. 2017. Improved training of wasserstein gans. In Advances in Neural Information Processing Systems. 5767--5777.

[7]

Tal Hassner, Shai Harel, Eran Paz, and Roee Enbar. 2015. Effective face frontalization in unconstrained images. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 4295--4304.

[8]

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition. 770--778.

[9]

Rui Huang, Shu Zhang, Tianyu Li, and Ran He. 2017. Beyond face rotation: Global and local perception gan for photorealistic and identity preserving frontal view synthesis. In Proceedings of the IEEE International Conference on Computer Vision. 2439--2448.

[10]

Xun Huang, Ming-Yu Liu, Serge Belongie, and Jan Kautz. 2018. Multimodal unsupervised image-to-image translation. In Proceedings of the European Conference on Computer Vision (ECCV). 172--189.

Digital Library

[11]

Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition . 1125--1134.

[12]

Yu-Gang Jiang, Minjun Li, Xi Wang, Wei Liu, and Xian-Sheng Hua. 2018a. DeepProduct: Mobile product search with portable deep features. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Vol. 14, 2 (2018), 50.

Digital Library

[13]

Yu-Gang Jiang, Zuxuan Wu, Jinhui Tang, Zechao Li, Xiangyang Xue, and Shih-Fu Chang. 2018b. Modeling multimodal clues in a hybrid deep learning framework for video classification. IEEE Transactions on Multimedia, Vol. 20, 11 (2018), 3137--3147.

Digital Library

[14]

Meina Kan, Shiguang Shan, Hong Chang, and Xilin Chen. 2014. Stacked progressive auto-encoders (spae) for face recognition across poses. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 1883--1890.

Digital Library

[15]

Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. 2017. Progressive growing of gans for improved quality, stability, and variation. arXiv preprint arXiv:1710.10196 (2017).

[16]

Tero Karras, Samuli Laine, and Timo Aila. 2018. A style-based generator architecture for generative adversarial networks. arXiv preprint arXiv:1812.04948 (2018).

[17]

Oliver Langner, Ron Dotsch, Gijsbert Bijlstra, Daniel HJ Wigboldus, Skyler T Hawk, and AD Van Knippenberg. 2010. Presentation and validation of the Radboud Faces Database. Cognition and emotion, Vol. 24, 8 (2010), 1377--1388.

[18]

Minjun Li, Haozhi Huang, Lin Ma, Wei Liu, Tong Zhang, and Yugang Jiang. 2018. Unsupervised Image-to-Image Translation with Stacked Cycle-Consistent Adversarial Networks. In Proceedings of the European Conference on Computer Vision (ECCV). 184--199.

[19]

Shaoxin Li, Xin Liu, Xiujuan Chai, Haihong Zhang, Shihong Lao, and Shiguang Shan. 2012. Morphable displacement field based image matching for face recognition across pose. In European conference on computer vision. Springer, 102--115.

Digital Library

[20]

Bei Liu, Jianlong Fu, Makoto P Kato, and Masatoshi Yoshikawa. 2018. Beyond narrative description: generating poetry from images by multi-adversarial training. In 2018 ACM Multimedia Conference on Multimedia Conference. ACM, 783--791.

Digital Library

[21]

Christos Sagonas, Yannis Panagakis, Stefanos Zafeiriou, and Maja Pantic. 2015. Robust statistical face frontalization. In Proceedings of the IEEE international conference on computer vision. 3871--3879.

Digital Library

[22]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 815--823.

[23]

Soumyadip Sengupta, Jun-Cheng Chen, Carlos Castillo, Vishal M Patel, Rama Chellappa, and David W Jacobs. 2016. Frontal to profile face verification in the wild. In 2016 IEEE Winter Conference on Applications of Computer Vision (WACV). IEEE, 1--9.

[24]

Yujun Shen, Ping Luo, Junjie Yan, Xiaogang Wang, and Xiaoou Tang. 2018. FaceID-GAN: Learning a symmetry three-player GAN for identity-preserving face synthesis. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 821--830.

[25]

Luan Tran, Xi Yin, and Xiaoming Liu. 2017. Disentangled representation learning gan for pose-invariant face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 1415--1424.

[26]

Jimei Yang, Scott E Reed, Ming-Hsuan Yang, and Honglak Lee. 2015. Weakly-supervised disentangling with recurrent transformations for 3d view synthesis. In Advances in Neural Information Processing Systems. 1099--1107.

[27]

Dong Yi, Zhen Lei, Shengcai Liao, and Stan Z Li. 2014. Learning face representation from scratch. arXiv preprint arXiv:1411.7923 (2014).

[28]

Junho Yim, Heechul Jung, ByungIn Yoo, Changkyu Choi, Dusik Park, and Junmo Kim. 2015. Rotating your face using multi-task deep neural network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 676--684.

[29]

Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. 2017. Towards large-pose face frontalization in the wild. In Proceedings of the IEEE International Conference on Computer Vision. 3990--3999.

[30]

Jichao Zhang, Yezhi Shu, Songhua Xu, Gongze Cao, Fan Zhong, and Xueying Qin. 2018. Sparsely Grouped Multi-task Generative Adversarial Networks for Facial Attribute Manipulation. arXiv preprint arXiv:1805.07509 (2018).

[31]

Yizhe Zhang, Ming Shao, Edward K Wong, and Yun Fu. 2013. Random faces guided sparse many-to-one encoder for pose-invariant face recognition. In Proceedings of the IEEE International Conference on Computer Vision . 2416--2423.

Digital Library

[32]

Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. 2017. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision. 2223--2232.

[33]

Xiangyu Zhu, Zhen Lei, Junjie Yan, Dong Yi, and Stan Z Li. 2015. High-fidelity pose and expression normalization for face recognition in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition . 787--796.

[34]

Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2013. Deep learning identity-preserving face space. In Proceedings of the IEEE International Conference on Computer Vision. 113--120.

Digital Library

[35]

Zhenyao Zhu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. 2014. Multi-view perceptron: a deep model for learning face identity and view representations. In Advances in Neural Information Processing Systems. 217--225.

Cited By

Sharma ASharma GAsiri FBhutto JBarnawi A(2024)Optimized Mirror Generative Adversarial Network with BERT Neural Architecture for Text Caption to Image ConversionSN Computer Science10.1007/s42979-024-02609-75:4Online publication date: 27-Mar-2024
https://doi.org/10.1007/s42979-024-02609-7
Kammoun ASlama RTabia HOuni TAbid M(2023)FNR-GAN: Face Normalization and Recognition with Generative Adversarial NetworksImage and Vision Computing10.1007/978-3-031-25825-1_10(131-143)Online publication date: 4-Feb-2023
https://doi.org/10.1007/978-3-031-25825-1_10
Chen JZhang JSangineto EChen TFan JSebe N(2021)Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00371(3664-3673)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00371
Show More Cited By

Index Terms

TC-GAN: Triangle Cycle-Consistent GANs for Face Frontalization with Facial Features Preserved
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
  2. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Lighting-aware face frontalization for unconstrained face recognition

Provide both lighting-recovered and lighting-normalized frontalized images.Basic frontalization with a generic 3D face model by the alignment of only five landmarks.Lighting recovered and normalized image filling by the symmetry of quotient image.LRFF ...
Read More
Facial Expression-Aware Face Frontalization
Computer Vision – ACCV 2016
Abstract
Face frontalization is a rising technique for view-invariant face analysis. It enables a non-frontal facial image to recover its general facial appearances to frontal view. A few pioneering works have been proposed very recently. However, face ...
Read More
Deep Appearance Models: A Deep Boltzmann Machine Approach for Face Modeling

The "interpretation through synthesis" approach to analyze face images, particularly Active Appearance Models (AAMs) method, has become one of the most successful face modeling approaches over the last two decades. AAM models have ability to represent ...
Read More

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

MM '19: Proceedings of the 27th ACM International Conference on Multimedia

October 2019

2794 pages

ISBN:9781450368896

DOI:10.1145/3343031

General Chairs:
Laurent Amsaleg
CNRS-IRISA, France
,
Benoit Huet
EURECOM, France
,
Martha Larson
Radboud University and TU Delft (Netherlands)
,
Program Chairs:
Guillaume Gravier
CNRS-IRISA, France
,
Hayley Hung
Delft University of Technology Netherlands
,
Chong-Wah Ngo
City University of Hong Kong Hong Kong
,
Wei Tsang Ooi
National University of Singapore Singapore

Copyright © 2019 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGMM: ACM Special Interest Group on Multimedia

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 15 October 2019

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

MM '19

Sponsor:

SIGMM

MM '19: The 27th ACM International Conference on Multimedia

October 21 - 25, 2019

Nice, France

Acceptance Rates

MM '19 Paper Acceptance Rate 252 of 936 submissions, 27%;

Overall Acceptance Rate 995 of 4,171 submissions, 24%

Upcoming Conference

MM '24

Sponsor:
sigmm

The 32nd ACM International Conference on Multimedia

October 28 - November 1, 2024

Melbourne , VIC , Australia

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
536
Total Downloads

Downloads (Last 12 months)10
Downloads (Last 6 weeks)0

Other Metrics

View Author Metrics

Citations

Cited By

Sharma ASharma GAsiri FBhutto JBarnawi A(2024)Optimized Mirror Generative Adversarial Network with BERT Neural Architecture for Text Caption to Image ConversionSN Computer Science10.1007/s42979-024-02609-75:4Online publication date: 27-Mar-2024
https://doi.org/10.1007/s42979-024-02609-7
Kammoun ASlama RTabia HOuni TAbid M(2023)FNR-GAN: Face Normalization and Recognition with Generative Adversarial NetworksImage and Vision Computing10.1007/978-3-031-25825-1_10(131-143)Online publication date: 4-Feb-2023
https://doi.org/10.1007/978-3-031-25825-1_10
Chen JZhang JSangineto EChen TFan JSebe N(2021)Coarse-to-Fine Gaze Redirection with Numerical and Pictorial Guidance2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00371(3664-3673)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00371
Cao DZeng YWei XNie LHong RQin ZWen Chen CCucchiara RHua XQi GRicci EZhang ZZimmermann R(2020)Adversarial Video Moment Retrieval by Jointly Modeling Ranking and LocalizationProceedings of the 28th ACM International Conference on Multimedia10.1145/3394171.3413841(898-906)Online publication date: 12-Oct-2020
https://dl.acm.org/doi/10.1145/3394171.3413841

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents