research-article

3D Face Recognition using Photometric Stereo and Deep Learning

Authors:

Wenhao ZhangAuthors Info & Claims

WIMS 2020: Proceedings of the 10th International Conference on Web Intelligence, Mining and Semantics

Pages 255 - 261

https://doi.org/10.1145/3405962.3405995

Published: 24 August 2020 Publication History

Abstract

Illumination variance is one of the largest real-world problems when deploying face recognition systems. Over the last few years much work has gone into the development of novel 3D face recognition methods to overcome this issue. Photometric stereo is a well-established 3D reconstruction technique capable of recovering the normals and albedo of a surface. Although it provides a way to obtain 3D data, the amount of training data available captured using photometric stereo often does not provide sufficient modelling capacity for training state-of-the-art feature extractors, such as deep convolutional neural networks, from scratch.

In this work we present a novel approach to utilising the lighting apparatus commonly used for photometric stereo to synthesise data that can act as a biometric. Combining this with deep learning techniques not only did we achieve near state-of-the-art results, but it gave insight into the possibility of using photometric stereo without the need of reconstruction. This could not only simplify the face recognition process but avoid unnecessary error that may arise from reconstruction.

Additionally, we utilise the active lighting from photometric stereo to evaluate the effect of illumination on face recognition. We compare our method to the state-of-the-art 3D methods and discuss potential use cases for our system.

References

[1]

Woodham, R. J., Y. Iwahori, and Rob A. Barman.: Photometric stereo: Lambertian reflec-tance and light sources with unknown direction and strength. University of British Colum-bia. Department of Computer Science (1991)

[2]

Chang, Kyong I., Kevin W. Bowyer, and Patrick J. Flynn.: An evaluation of multimodal 2D+ 3D face biometrics. In: IEEE transactions on pattern analysis and machine intelligence 27, vol. 4: pp. 619--624 (2005)

[3]

Roy, Sébastien, and Ingemar J. Cox.: A maximum-flow formulation of the n-camera stereo correspondence problem. In: IEEE Sixth International Conference Computer Vision, pp. 492--499 (1998).

[4]

Roth, Joseph, Yiying Tong, and Xiaoming Liu.: Adaptive 3D face reconstruction from un-constrained photo collections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4197--4206 (2016)

[5]

Brown, Myron Z., Darius Burschka, and Gregory D. Hager.: Advances in computational stereo. In: IEEE transactions on pattern analysis and machine intelligence 25, vol. 8, pp. 993--1008 (2003)

[6]

Hansen, Mark F., Gary A. Atkinson, Lyndon N. Smith, and Melvyn L. Smith.: 3D face re-constructions from photometric stereo using near infrared and visible light. In: Computer Vi-sion and Image Understanding 114, vol. 8 pp. 942--951 (2010)

[7]

Sun, Yujuan, Junyu Dong, Muwei Jian, and Lin Qi.: Fast 3D face reconstruction based on uncalibrated photometric stereo. In: Multimedia Tools and Applications 74, vol. 11, pp. 3635--3650 (2015)

Digital Library

[8]

Chang, K. B. K. I., Kevin Bowyer, and Patrick Flynn.: Face recognition using 2D and 3D facial data. In: ACM Workshop on Multimodal User Authentication, pp. 25--32 (2003)

[9]

Sabater, Neus, J-M. Morel, and Andrés Almansa.: How accurate can block matches be in stereo vision?. In: SIAM Journal on Imaging Sciences 4, vol. 1, pp. 472--500 (2011)

[10]

Sellahewa, Harin, and Sabah A. Jassim.: Image-quality-based adaptive face recognition. In: IEEE Transactions on Instrumentation and measurement 59, vol. 4, pp. 805--813 (2010)

[11]

Brahnam, Sheryl.: Local binary patterns: new variants and applications. Springer Berlin Hei-delberg (2016)

[12]

Yang, Meng, and Lei Zhang.: Gabor feature based sparse representation for face recognition with gabor occlusion dictionary. In: European conference on computer vision, pp. 448--461 (2010)

[13]

Jing, Xiao-Yuan, David Zhang, and Yuan-Yan Tang.: An improved LDA approach. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics) 34, vol. 5, pp. 1942--1951 (2004)

[14]

Kautkar, Satyajit N., Gary A. Atkinson, and Melvyn L. Smith.: Face recognition in 2D and 2.5 D using ridgelets and photometric stereo. In: Pattern recognition 45, vol. 9, pp. 3317--3327 (2012)

[15]

Villarini, Barbara, Athanasios Gkelias, and Vasilios Argyriou.: Photometric Stereo for 3D face reconstruction using non linear illumination models. In: IAPR Workshop on Multimod-al Pattern Recognition of Social Signals in Human-Computer Interaction, pp. 140--152 (2016)

[16]

Lee, Sang-Woong, Patrick SP Wang, Svetlana N. Yanushkevich, and Seong-Whan Lee.: Noniterative 3D face reconstruction based on photometric stereo. International Journal of Pattern Recognition and Artificial Intelligence 22, vol. 03, pp. 389--410 (2008)

[17]

Lao, Shihong, Yasushi Sumi, Masato Kawade, and Fumiaki Tomita.: 3D template matching for pose invariant face recognition using 3D facial model built with isoluminance line based stereo vision. In: Proceedings 15th International Conference on Pattern Recognition, vol. 2, pp. 911--916 (2000)

[18]

Georghiades, Athinodoros S., Peter N. Belhumeur, and David J. Kriegman.: From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE transactions on pattern analysis and machine intelligence 23, vol. 6, pp. 643--660 (2001)

[19]

Hansen, Mark F., Gary A. Atkinson, and Melvyn L. Smith.: BRDF estimation for faces from a sparse dataset using a neural network. In: International Conference on Computer Analysis of Images and Patterns, pp. 212--220 (2013)

Digital Library

[20]

Kee, Seok Cheol, Kyoung Mu Lee, and Sang Uk Lee.: Illumination invariant face recogni-tion using photometric stereo. IEICE TRANSACTIONS on Information and Systems 83, vol. 7, pp. 1466--1474 (2000)

[21]

Abate, Andrea F., Michele Nappi, Daniel Riccio, and Gabriele Sabatino.: 2D and 3D face recognition: A survey. Pattern recognition letters 28, vol. 14, pp. 1885--1906 (2007)

[22]

Paysan, Pascal, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.: A 3D face model for pose and illumination invariant face recognition. In: AVSS'09. Sixth IEEE International Conference on Advanced video and signal based surveillance, pp. 296--301 (2009)

Digital Library

[23]

Soltana, W. Ben, Di Huang, Mohsen Ardabilian, Liming Chen, and C. Ben Amar.: Compar-ison of 2D/3D features and their adaptive score level fusion for 3D face recognition. In: 3D Data Processing, Visualization and Transmission (2010)

[24]

Adini, Yael, Yael Moses, and Shimon Ullman.: Face recognition: The problem of compen-sating for changes in illumination direction. In: IEEE Transactions on pattern analysis and machine intelligence 19, vol. 7, pp. 721--732 (1997)

Digital Library

[25]

Zhao, Wenyi, Rama Chellappa, P. Jonathon Phillips, and Azriel Rosenfeld.: Face recogni-tion: A literature survey. ACM computing surveys (CSUR) 35, vol. 4, pp. 399--458 (2003)

[26]

Baltrušaitis, Tadas, Peter Robinson, and Louis-Philippe Morency.: Openface: an open source facial behavior analysis toolkit. In: Applications of Computer Vision (WACV), 2016 IEEE Winter Conference on, pp. 1--10 (2016)

[27]

Schroff, Florian, Dmitry Kalenichenko, and James Philbin.: Facenet: A unified embedding for face recognition and clustering. In: Proceedings of the IEEE conference on computer vi-sion and pattern recognition, pp. 815--823 (2015)

[28]

Viola, Paul, and Michael J. Jones.: Robust real-time face detection. International journal of computer vision 57, vol. 2, pp. 137--154 (2004)

[29]

Zafeiriou, Stefanos, Mark Hansen, Gary Atkinson, Vasileios Argyriou, Maria Petrou, Melvyn Smith, and Lyndon Smith. The photoface database. In: Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on, pp. 132--139 (2011)

[30]

VGGFace Homepage, http://www.robots.ox.ac.uk/~vgg/software/vgg_face/, last accessed 2019/02/03

[31]

Parkhi, Omkar M., Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In: BMVC, vol. 1, pp. 6 (2015)

[32]

LFW Dataset Homepage, http://vis-www.cs.umass.edu/lfw/, last accessed 2019/01/19

[33]

YouTube Faces Dataset Homepage, https://www.cs.tau.ac.il/~wolf/ytfaces/, last accessed 2019/01/17

[34]

Zafeiriou, Stefanos, Gary A. Atkinson, Mark F. Hansen, William AP Smith, Vasileios Ar-gyriou, Maria Petrou, Melvyn L. Smith, and Lyndon N. Smith.: Face recognition and verifi-cation using photometric stereo: The photoface database and a comprehensive evaluation. IEEE transactions on information forensics and security 8, vol. 1 pp. 121--135 (2013)

[35]

Kasturi, Rangachar, Dmitry Goldgof, Padmanabhan Soundararajan, Vasant Manohar, John Garofolo, Rachel Bowers, Matthew Boonstra, Valentina Korzhova, and Jing Zhang.: Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics, and protocol. IEEE Transactions on Pattern Analysis and Machine In-telligence 31, vol. 2, pp. 319--336 (2009)

[36]

Huang, Shih-Ming, and Jar-Ferr Yang. Improved principal component regression for face recognition under illumination variations. IEEE signal processing letters 19, vol. 4, pp. 179--182 (2012)

[37]

Paysan, Pascal, Reinhard Knothe, Brian Amberg, Sami Romdhani, and Thomas Vetter.: A 3D face model for pose and illumination invariant face recognition. In: Sixth IEEE Interna-tional Conference on Advanced Video and Signal Based Surveillance, pp. 296--301 (2009)

Digital Library

[38]

Turk, Matthew A., and Alex P. Pentland. Face recognition using eigenfaces. In: Proceedings IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pp. 586--591 (1991)

Cited By

Bhavani SKarthikeyan C(2023)Robust 3D face recognition in unconstrained environment using distance based ternary search siamese networkMultimedia Tools and Applications10.1007/s11042-023-17545-6Online publication date: 15-Nov-2023
https://doi.org/10.1007/s11042-023-17545-6
Samatas GPapakostas G(2022)Biometrics: Going 3DSensors10.3390/s2217636422:17(6364)Online publication date: 24-Aug-2022
https://doi.org/10.3390/s22176364

Index Terms

3D Face Recognition using Photometric Stereo and Deep Learning
1. Computer systems organization
  1. Real-time systems
    1. Real-time system specification
2. Computing methodologies
  1. Machine learning
    1. Machine learning approaches
      1. Neural networks

Recommendations

Interreflection removal for photometric stereo by using spectrum-dependent albedo
CVPR '11: Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition

We present a novel method that can separate m-bounced light and remove the interreflections in a photometric stereo setup. Under the assumption of a uniformly colored lambertian surface, the intensity of a point in the scene is the sum of 1-bounced ...
Fast 3D face reconstruction based on uncalibrated photometric stereo

This paper proposes a fast algorithm for three-dimensional face reconstruction using uncalibrated Photometric Stereo. With a reference face model, lighting parameters are estimated from input face images lighted by unknown illumination, which can be ...
Face recognition in 2D and 2.5D using ridgelets and photometric stereo

A new technique for face recognition - Ridgefaces - is presented. The method combines the well-known Fisherface method with the ridgelet transform and high-speed Photometric Stereo (PS). The paper first derives ridgelet projections for 2D/2.5D face ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

WIMS 2020: Proceedings of the 10th International Conference on Web Intelligence, Mining and Semantics

June 2020

279 pages

ISBN:9781450375429

DOI:10.1145/3405962

Conference Chairs:
Richard Chbeir
University Pau & Pays Adour
,
Yannis Manolopoulos
Open University of Cyprus, Cyprus
,
Rajendra Akerkar
Western Norway Research Institute, Norway
,
Jolanta Mizera-Pietraszko
Military University of Land Forces, Poland

Copyright © 2020 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 24 August 2020

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

WIMS 2020

WIMS 2020: The 10th International Conference on Web Intelligence, Mining and Semantics

June 30 - July 3, 2020

Biarritz, France

Acceptance Rates

WIMS 2020 Paper Acceptance Rate 35 of 63 submissions, 56%;

Overall Acceptance Rate 140 of 278 submissions, 50%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
109
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)0

Reflects downloads up to 17 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Bhavani SKarthikeyan C(2023)Robust 3D face recognition in unconstrained environment using distance based ternary search siamese networkMultimedia Tools and Applications10.1007/s11042-023-17545-6Online publication date: 15-Nov-2023
https://doi.org/10.1007/s11042-023-17545-6
Samatas GPapakostas G(2022)Biometrics: Going 3DSensors10.3390/s2217636422:17(6364)Online publication date: 24-Aug-2022
https://doi.org/10.3390/s22176364

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents