Abstract
In this paper we show how to estimate facial surface reflectance properties (a slice of the BRDF and the albedo) in conjunction with the facial shape from a single image. The key idea underpinning our approach is to iteratively interleave the two processes of estimating reflectance properties based on the current shape estimate and updating the shape estimate based on the current estimate of the reflectance function. For frontally illuminated faces, the reflectance properties can be described by a function of one variable which we estimate by fitting a curve to the scattered and noisy reflectance samples provided by the input image and estimated shape. For non-frontal illumination, we fit a smooth surface to the scattered 2D reflectance samples. We make use of a novel statistical face shape constraint which we term ‘model-based integrability’ which we use to regularise the shape estimation. We show that the method is capable of recovering accurate shape and reflectance information from single grayscale or colour images using both synthetic and real world imagery. We use the estimated reflectance measurements to render synthetic images of the face in varying poses. To synthesise images under novel illumination, we show how to fit a parametric model of reflectance to the estimated reflectance function.
Similar content being viewed by others
References
Ahmed, A. H., & Farag, A. A. (2006). A new formulation for shape from shading for non–Lambertian surfaces. In Proc. CVPR (Vol. 2, pp. 1817–1824).
Atick, J. J., Griffin, P. A., & Redlich, A. N. (1996). Statistical approach to SFS: Reconstruction of 3D face surfaces from single 2D images. Neural Computation, 8(6), 1321–1340.
Atkinson, G., & Hancock, E. R. (2008). Two-dimensional BRDF estimation from polarisation. Computer Vision and Image Understanding, 111(2), 126–141.
Baranoski, G. V. G., & Krishnaswamy, A. (2004). An introduction to light interaction with human skin. Revista de Informá, tica Teórica e Aplicada, XI(1), 33–62.
Basri, R., & Jacobs, D. W. (2003). Lambertian reflectance and linear subspaces. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(2), 218–233.
Blanz, V., & Vetter, T. (2003). Face recognition based on fitting a 3D morphable model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(9), 1063–1074.
Claridge, E., Cotton, S., Hall, P., & Moncrieff, M. (2003). From colour to tissue histology: physics based interpretation of images of pigmented skin lesions. Medical Image Analysis Journal, 7, 489–502.
Cootes, T. F., Edwards, G. J., & Taylor, C. J. (1998). Active appearance models. In Proc. ECCV (pp. 484–498).
Dana, K. J., van Ginneken, B., Nayar, S. K., & Koenderink, J. J. (1999). Reflectance and texture of real–world surfaces. ACM Transactions on Graphics, 18(1), 1–34.
Debevec, P., Hawkins, T., Tchou, C., Duiker, H.-P., & Sarokin, W. (2000). Acquiring the reflectance field of a human face. In Proc. SIGGRAPH (2000).
D’Errico, J. (2005). Surface fitting using gridfit. MATLAB Central File Exchange.
Donner, C., & Wann Jensen, H. (2006). A spectral BSSRDF for shading human skin. In Rendering techniques 2006: 17th eurographics workshop on rendering (pp. 409–418).
Dovgard, R., & Basri, R. (2004). Statistical symmetric shape from shading for 3D structure recovery of faces. In Proc. ECCV (Vol. 2, pp. 99–113).
Durou, J.-D., Falcone, M., & Sagona, M. (2008). Numerical methods for shape-from-shading: A new survey with benchmarks. Computer Vision and Image Understanding, 109(1), 22–43.
Frankot, R. T., & Chellappa, R. (1988). A method for enforcing integrability in shape from shading algorithms. IEEE Transactions on Pattern Analysis and Machine Intelligence, 10(4), 439–451.
Fuchs, M., Blanz, V., Lensch, H., & Seidel, H.-P. (2005). Reflectance from images: A model–based approach for human faces. IEEE Transactions on Visualization and Computer Graphics, 11(3), 296–305.
Georghiades, A. (2003). Recovering 3–d shape and reflectance from a small number of photographs. In Eurographics symposium on rendering (pp. 230–240).
Georghiades, A. S., Belhumeur, P. N., & Kriegman, D. J. (2001). From few to many: Illumination cone models for face recognition under variable lighting and pose. IEEE Transactions on Pattern Analysis and Machine Intelligence, 23(6), 643–660.
Hertzmann, A., & Seitz, S. M. (2005). Example–based photometric stereo: Shape reconstruction with general, varying BRDFs. IEEE Transactions on Pattern Analysis and Machine Intelligence, 27(8), 1254–1264.
Horn, B. K. P., & Sjoberg, R. W. (1979). Calculating the reflectance map. Applied Optics, 18(11), 1770–1779.
Kemelmacher, I., & Basri, R. (2006). Molding face shapes by example. In Proc. ECCV (pp. 277–288).
Koenderink, J., & Pont, S. (2003). The secret of velvety skin. Machine Vision and Applications, 14, 260–268.
Marschner, S. R., Westin, S. H., Lafortune, E. P. F., Torrance, K. E., & Greenberg, D. P. (1999). Reflectance measurements of human skin (Technical Report PCG–99–2). Cornell University.
Oliensis, J. (1991). Uniqueness in shape from shading. International Journal of Computer Vision, 6(2), 75–104.
Oren, M., & Nayar, S. K. (1995). Generalization of the Lambertian model and implications for machine vision. International Journal of Computer Vision, 14(3), 227–251.
Phong, B. T. (1975). Illumination for computer generated images. Communications of the ACM, 18(6), 311–317.
Prados, E., & Faugeras, O. (2005). A generic and provably convergent shape-from-shading method for orthographic and pinhole cameras. International Journal of Computer Vision, 65(1–2), 97–125.
Ragheb, H., & Hancock, E. R. (2003). A probabilistic framework for specular shape–from–shading. Pattern Recognition, 36(2), 407–427.
Robles-Kelly, A., & Hancock, E. R. (2005). Estimating the surface radiance function from single images. Graphical Models, 67(6), 518–548.
Sim, T., Baker, S., & Bsat, M. (2003). The CMU pose, illumination, and expression database. IEEE Transactions on Pattern Analysis and Machine Intelligence, 25(12), 1615–1618.
Smith, W. A. P., & Hancock, E. R. (2007). A new framework for grayscale and colour non-Lambertian shape-from-shading. In Proc. ACCV (pp. 869–880).
Smith, W. A. P., & Hancock, E. R. (2008). Facial shape–from–shading and recognition using principal geodesic analysis and robust statistics. International Journal of Computer Vision, 76(1), 71–91.
Torrance, K., & Sparrow, E. (1967). Theory for off–specular reflection from roughened surfaces. Journal of the Optical Society of America, 57(9), 1105–1114.
USF HumanID 3D Face Database (1998). Courtesy of Sudeep Sarkar, University of South Florida, Tampa, FL.
van Gemert, M. J. C., Jacques, S. L., Sterenborg, H. J. C. M., & Star, W. M. (1989). Skin optics. IEEE Transactions on Biomedical Engineering, 36(12), 1146–1154.
Weyrich, T., Matusik, W., Pfister, H., Bickel, B., Donner, C., Tu, C., McAndless, J., Lee, J., Ngan, A., Wann Jensen, H., & Gross, M. (2006). Analysis of human faces using a measurement-based skin reflectance model. ACM Transactions on Graphics, 25(3), 1013–1024.
Zhang, L., & Samaras, D. (2006). Face recognition from a single training image under arbitrary unknown lighting using spherical harmonics. IEEE Transactions on Pattern Analysis and Machine Intelligence, 28(3), 351–363.
Zhang, R., Tsai, P. S., Cryer, J. E., & Shah, M. (1999). Shape–from–shading: a survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 21(8), 690–706.
Zhao, W. Y., & Chellappa, R. (2001). Symmetric shape–from–shading using self–ratio image. International Journal of Computer Vision, 45, 55–75.
Zickler, T., Mallick, S. P., Kriegman, D. J., & Belhumeur, P. N. (2008). Color subspaces as photometric invariants. International Journal of Computer Vision, 79(1), 13–30
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Smith, W.A.P., Hancock, E.R. Estimating Facial Reflectance Properties Using Shape-from-Shading. Int J Comput Vis 86, 152–170 (2010). https://doi.org/10.1007/s11263-008-0175-z
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11263-008-0175-z