A Self-Occlusion Aware Lighting Model for Real-Time Dynamic Reconstruction
Authors: 
Chengwei Zheng, Wenbin Lin, Feng XuAuthors Info & Claims
Chengwei Zheng, Wenbin Lin, Feng XuAuthors Info & ClaimsPages 4062 - 4073
Abstract
In real-time dynamic reconstruction, geometry and motion are the major focuses while appearance is not fully explored, leading to the low-quality appearance of the reconstructed surfaces. In this article, we propose a lightweight lighting model that considers spatially varying lighting conditions caused by self-occlusion. This model estimates per-vertex masks on top of a single Spherical Harmonic (SH) lighting to represent spatially varying lighting conditions without adding too much computation cost. The mask is estimated based on the local geometry of a vertex to model the self-occlusion effect, which is the major reason leading to the spatial variation of lighting. Furthermore, to use this model in dynamic reconstruction, we also improve the motion estimation quality by adding a real-time per-vertex displacement estimation step. Experiments demonstrate that both the reconstructed appearance and the motion are largely improved compared with the current state-of-the-art techniques.
References
[1]
R. A. Newcombe, D. Fox, and S. M. Seitz, “DynamicFusion: Reconstruction and tracking of non-rigid scenes in real-time,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2015, pp. 343–352.
[2]
M. Innmann, M. Zollhöfer, M. Nießner, C. Theobalt, and M. Stamminger, “VolumeDeform: Real-time volumetric non-rigid reconstruction,” in Proc. Eur. Conf. Comput. Vis., 2016, pp. 362–379.
[3]
K. Guo, F. Xu, T. Yu, X. Liu, Q. Dai, and Y. Liu, “Real-time geometry, Albedo, and motion reconstruction using a single RGB-D camera,” ACM Trans. Graph., vol. 36, no. 4, 2017, Art. no.
[4]
P. Debevec, T. Hawkins, C. Tchou, H.-P. Duiker, W. Sarokin, and M. Sagar, “Acquiring the reflectance field of a human face,” in Proc. 27th Annu. Conf. Comput. Graph. Interactive Techn., 2000, pp. 145–156.
[5]
O. Alexander et al., “The digital emily project: Achieving a photorealistic digital actor,” IEEE Comput. Graphics Appl., vol. 30, no. 4, pp. 20–31, Jul./Aug. 2010.
[6]
P. Gotardo, J. Riviere, D. Bradley, A. Ghosh, and T. Beeler, “Practical dynamic facial appearance modeling and acquisition,” ACM Trans. Graph., vol. 37, no. 6, pp. 1–13, 2018.
[7]
M. Dou et al., “Motion2fusion: Real-time volumetric performance capture,” ACM Trans. Graph., vol. 36, no. 6, pp. 1–16, 2017.
[8]
R. Du, M. Chuang, W. Chang, H. Hoppe, and A. Varshney, “Montage4D: Interactive seamless fusion of multiview video textures,” in Proc. ACM SIGGRAPH Symp. Interactive 3D Graph. Games, 2018, pp. 1–11.
[9]
K. Guo et al., “The relightables: Volumetric performance capture of humans with realistic relighting,” ACM Trans. Graph., vol. 38, no. 6, pp. 1–19, 2019.
[10]
C. Zheng and F. Xu, “DTexFusion: Dynamic texture fusion using a consumer RGBD sensor,” IEEE Trans. Vis. Comput. Graphics, to be published.
[11]
T. Tung, S. Nobuhara, and T. Matsuyama, “Simultaneous super-resolution and 3D video using graph-cuts,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2008, pp. 1–8.
[12]
D. Casas, M. Volino, J. Collomosse, and A. Hilton, “4D video textures for interactive character appearance,” Comput. Graph. Forum, vol. 33, no. 2, pp. 371–380, 2014.
[13]
F. Prada, M. Kazhdan, M. Chuang, A. Collet, and H. Hoppe, “Spatiotemporal atlas parameterization for evolving meshes,” ACM Trans. Graph., vol. 36, no. 4, pp. 1–12, 2017.
[14]
A. E. Ichim, S. Bouaziz, and M. Pauly, “Dynamic 3D avatar creation from hand-held video input,” ACM Trans. Graph., vol. 34, no. 4, pp. 1–14, 2015.
[15]
P. Garrido et al., “Reconstruction of personalized 3D face rigs from monocular video,” ACM Trans. Graph., vol. 35, no. 3, pp. 1–15, 2016.
[16]
J. Thies, M. Zollhofer, M. Stamminger, C. Theobalt, and M. Nießner, “Face2Face: Real-time face capture and reenactment of RGB videos,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2016, pp. 2387–2395.
[17]
S. Saito, L. Wei, L. Hu, K. Nagano, and H. Li, “Photorealistic facial texture inference using deep neural networks,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 2326–2335.
[18]
K. Nagano et al., “paGAN: Real-time avatars using dynamic textures,” in Proc. SIGGRAPH Asia Tech. Papers, 2018, Art. no.
[19]
C. Wu, T. Shiratori, and Y. Sheikh, “Deep incremental learning for efficient high-fidelity face tracking,” in Proc. SIGGRAPH Asia Tech. Papers, 2018, Art. no.
[20]
S. Lombardi, J. Saragih, T. Simon, and Y. Sheikh, “Deep appearance models for face rendering,” ACM Trans. Graph., vol. 37, no. 4, pp. 1–13, 2018.
[21]
S.-E. Wei et al., “VR facial animation via multiview image translation,” ACM Trans. Graph., vol. 38, no. 4, 2019, Art. no.
[22]
R. Martin-Brualla et al., “LookinGood: Enhancing performance capture with real-time neural re-rendering,” ACM Trans. Graph., vol. 37, no. 6, pp. 1–14, 2018.
[23]
R. Pandey et al., “Volumetric capture of humans with a single RGBD camera via semi-parametric learning,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2019, pp. 9701–9710.
[24]
C. Rother, M. Kiefel, L. Zhang, B. Schölkopf, and P. Gehler, “Recovering intrinsic images with a global sparsity prior on reflectance,” in Proc. Int. Conf. Neural Inf. Process. Syst., 2011, pp. 765–773.
[25]
L. Shen and C. Yeo, “Intrinsic images decomposition using a local and global sparse representation of reflectance,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2011, pp. 697–704.
[26]
A. Bousseau, S. Paris, and F. Durand, “User assisted intrinsic images,” ACM Trans. Graph., vol. 28, no. 5, 2009, Art. no.
[27]
R. Grosse, M. K. Johnson, E. H. Adelson, and W. T. Freeman, “Ground truth dataset and baseline evaluations for intrinsic image algorithms,” in Proc. IEEE 12th Int. Conf. Comput. Vis., 2009, pp. 2335–2342.
[28]
E. Garces, A. Munoz, J. Lopez-Moreno, and D. Gutierrez, “Intrinsic images by clustering,” Comput. Graph. Forum, vol. 31, no. 4, pp. 1415–1424, 2012.
[29]
J. T. Barron and J. Malik, “Color constancy, intrinsic images, and shape estimation,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 57–70.
[30]
J. T. Barron and J. Malik, “Shape, albedo, and illumination from a single image of an unknown object,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2012, pp. 334–341.
[31]
S. Bi, X. Han, and Y. Yu, “An L1 image transform for edge-preserving smoothing and scene-level intrinsic decomposition,” ACM Trans. Graph., vol. 34, no. 4, pp. 1–12, 2015.
[32]
Z. Cheng, Y. Zheng, S. You, and I. Sato, “Non-local intrinsic decomposition with near-infrared priors,” in Proc. IEEE/CVF Int. Conf. Comput. Vis., 2019, pp. 2521–2530.
[33]
Q. Chen and V. Koltun, “A simple model for intrinsic image decomposition with depth cues,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 241–248.
[34]
J. Jeon, S. Cho, X. Tong, and S. Lee, “Intrinsic image decomposition using structure-texture separation and surface normals,” in Proc. Eur. Conf. Comput. Vis., 2014, pp. 218–233.
[35]
M. Hachama, B. Ghanem, and P. Wonka, “Intrinsic scene decomposition from RGB-D images,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 810–818.
[36]
X. Wei, G. Chen, Y. Dong, S. Lin, and X. Tong, “Object-based illumination estimation with rendering-aware neural networks,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 380–396.
[37]
G. Ye, E. Garces, Y. Liu, Q. Dai, and D. Gutierrez, “Intrinsic video and applications,” ACM Trans. Graph., vol. 33, no. 4, pp. 1–11, 2014.
[38]
N. Bonneel, K. Sunkavalli, J. Tompkin, D. Sun, S. Paris, and H. Pfister, “Interactive intrinsic video editing,” ACM Trans. Graph., vol. 33, no. 6, pp. 1–10, 2014.
[39]
A. Meka, M. Zollhöfer, C. Richardt, and C. Theobalt, “Live intrinsic video,” ACM Trans. Graph., vol. 35, no. 4, pp. 1–14, 2016.
[40]
A. Meka, G. Fox, M. Zollhöfer, C. Richardt, and C. Theobalt, “Live user-guided intrinsic video for static scenes,” IEEE Trans. Vis. Comput. Graphics, vol. 23, no. 11, pp. 2447–2454, Nov. 2017.
[41]
D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black, “A naturalistic open source movie for optical flow evaluation,” in Proc. Eur. Conf. Comput. Vis., 2012, pp. 611–625.
[42]
S. Bell, K. Bala, and N. Snavely, “Intrinsic images in the wild,” ACM Trans. Graph., vol. 33, no. 4, pp. 1–12, 2014.
[43]
A. X. Chang et al., “ShapeNet: An information-rich 3D model repository,” 2015,.
[44]
B. Kovacs, S. Bell, N. Snavely, and K. Bala, “Shading annotations in the wild,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 850–859.
[45]
Z. Li and N. Snavely, “CGIntrinsics: Better intrinsic image decomposition through physically-based rendering,” in Proc. Eur. Conf. Comput. Vis., 2018, pp. 371–387.
[46]
T. Zhou, P. Krahenbuhl, and A. A. Efros, “Learning data-driven reflectance priors for intrinsic image decomposition,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 3469–3477.
[47]
T. Narihira, M. Maire, and S. X. Yu, “Direct intrinsics: Learning albedo-shading decomposition by convolutional regression,” in Proc. IEEE Int. Conf. Comput. Vis., 2015, pp. 2992–2992.
[48]
J. Shi, Y. Dong, H. Su, and S. X. Yu, “Learning non-lambertian object intrinsics across ShapeNet categories,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2017, pp. 5844–5853.
[49]
Q. Fan, J. Yang, G. Hua, B. Chen, and D. Wipf, “Revisiting deep intrinsic image decompositions,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 8944–8952.
[50]
Z. Li and N. Snavely, “Learning intrinsic image decomposition from watching the world,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 9039–9048.
[51]
L. Cheng, C. Zhang, and Z. Liao, “Intrinsic image transformation via scale space decomposition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 656–665.
[52]
Y. Liu, Y. Li, S. You, and F. Lu, “Unsupervised learning for intrinsic image decomposition from a single image,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 3245–3254.
[53]
S. Sengupta, A. Kanazawa, C. D. Castillo, and D. W. Jacobs, “SfSNet: Learning shape, reflectance and illuminance of faces ‘in the wild’,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2018, pp. 6296–6305.
[54]
Y. Kanamori and Y. Endo, “Relighting humans: Occlusion-aware inverse rendering for full-body human images,” 2019,.
[55]
Y. Yu and W. A. P. Smith, “InverseRenderNet: Learning single image inverse rendering,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2019, pp. 3150–3159.
[56]
J. Luo, Z. Huang, Y. Li, X. Zhou, G. Zhang, and H. Bao, “NIID-Net: Adapting surface normal knowledge for intrinsic image decomposition in indoor scenes,” IEEE Trans. Vis. Comput. Graphics, vol. 26, no. 12, pp. 3434–3445, Dec. 2020.
[57]
B. K. Horn, “Shape from shading: A method for obtaining the shape of a smooth opaque object from one view,” 1970.
[58]
J. T. Barron and J. Malik, “Intrinsic scene properties from a single RGB-D image,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 17–24.
[59]
L.-F. Yu, S.-K. Yeung, Y.-W. Tai, and S. Lin, “Shading-based shape refinement of RGB-D images,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit., 2013, pp. 1415–1422.
[60]
C. Wu, M. Zollhöfer, M. Nießner, M. Stamminger, S. Izadi, and C. Theobalt, “Real-time shading-based refinement for consumer depth cameras,” ACM Trans. Graph., vol. 33, no. 6, pp. 1–10, 2014.
[61]
Y. Han, J.-Y. Lee, and I. So Kweon, “High quality shape from a single RGB-D image under uncalibrated natural illumination,” in Proc. IEEE Int. Conf. Comput. Vis., 2013, pp. 1617–1624.
[62]
X. Zuo, S. Wang, J. Zheng, and R. Yang, “Detailed surface geometry and albedo recovery from RGB-D video under natural illumination,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 3152–3161.
[63]
R. Maier, K. Kim, D. Cremers, J. Kautz, and M. Nießner, “Intrinsic3D: High-quality 3D reconstruction by joint appearance and geometry optimization with spatially-varying lighting,” in Proc. IEEE Int. Conf. Comput. Vis., 2017, pp. 3133–3141.
[64]
G. Xing, Y. Liu, H. Ling, X. Granier, and Y. Zhang, “Automatic spatially varying illumination recovery of indoor scenes based on a single RGB-D image,” IEEE Trans. Vis. Comput. Graphics, vol. 26, no. 4, pp. 1672–1685, Apr. 2020.
[65]
Z. Li, M. Shafiei, R. Ramamoorthi, K. Sunkavalli, and M. Chandraker, “Inverse rendering for complex indoor scenes: Shape, spatially-varying lighting and SVBRDF from a single image,” in Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit., 2020, pp. 2472–2481.
[66]
Y. Yu, A. Meka, M. Elgharib, H.-P. Seidel, C. Theobalt, and W. A. Smith, “Self-supervised outdoor scene relighting,” in Proc. Eur. Conf. Comput. Vis., 2020, pp. 84–101.
[67]
M. Knecht, “State of the art report on ambient occlusion,” Vienna Institute of Technology, 2007.
[68]
S. Zhukov, A. Iones, and G. Kronin, “An ambient light illumination model,” in Proc. Eurographics Workshop Rendering Techn., 1998, pp. 45–55.
[69]
F. Hernell, P. Ljung, and A. Ynnerman, “Local ambient occlusion in direct volume rendering,” IEEE Trans. Vis. Comput. Graphics, vol. 16, no. 4, pp. 548–559, Jul./Aug. 2010.
[70]
P. Shanmugam and O. Arikan, “Hardware accelerated ambient occlusion techniques on GPUs,” in Proc. Symp. Interactive 3D Graph. Games, 2007, pp. 73–80.
[71]
C. K. Reinbothe, T. Boubekeur, and M. Alexa, “Hybrid ambient occlusion,” in Proc. Annu. Conf. Eur. Assoc. Comput. Graph., 2009, pp. 51–57.
[72]
T. Ropinski, J. Meyer-Spradow, S. Diepenbrock, J. Mensmann, and K. Hinrichs, “Interactive volume rendering with dynamic ambient occlusion and color bleeding,” Comput. Graph. Forum, vol. 27, no. 2, pp. 567–576, 2008.
[73]
J. Kontkanen and S. Laine, “Ambient occlusion fields,” in Proc. Symp. Interactive 3D Graph. Games, 2005, pp. 41–48.
[74]
L. Bavoil, M. Sainz, and R. Dimitrov, “Image-space horizon-based ambient occlusion,” in Proc. ACM SIGGRAPH Talks, 2008, pp. 1–1.
[75]
J. Diaz, P.-P. Vazquez, I. Navazo, and F. Duguet, “Real-time ambient occlusion and halos with summed area tables,” Comput. Graph., vol. 34, no. 4, pp. 337–350, 2010.
[76]
S. Laine and T. Karras, “Two methods for fast ray-cast ambient occlusion,” Comput. Graph. Forum, vol. 29, no. 4, pp. 1325–1333, 2010.
[77]
D. Hauagge, S. Wehrwein, K. Bala, and N. Snavely, “Photometric ambient occlusion for intrinsic image decomposition,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 38, no. 4, pp. 639–651, Apr. 2016.
[78]
R. Ramamoorthi and P. Hanrahan, “An efficient representation for irradiance environment maps,” in Proc. 28th Annu. Conf. Comput. Graph. Interactive Techn., 2001, pp. 497–500.
[79]
M. Zollhöfer et al., “Shading-based refinement on volumetric signed distance functions,” ACM Trans. Graph., vol. 34, no. 4, pp. 1–14, 2015.
Recommendations
Real-time rendering of animated hair under dynamic, low-frequency environmental lighting
VRCAI '12: Proceedings of the 11th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in IndustryWe present a fast algorithm for rendering animated hair under a dynamic lighting environment at interactive frame rates. We use spherical harmonics (SH) to represent the environmental lighting. Since SH functions are orthogonal, the environmental light ...
Real-Time Volume-Based Ambient Occlusion
Real-time rendering can benefit from global illumination methods to make the 3D environments look more convincing and lifelike. On the other hand, the conventional global illumination algorithms for the estimation of the diffuse surface interreflection ...
Precomputed radiance transfer for real-time rendering in dynamic, low-frequency lighting environments
We present a new, real-time method for rendering diffuse and glossy objects in low-frequency lighting environments that captures soft shadows, interreflections, and caustics. As a preprocess, a novel global transport simulator creates functions over the ...
Comments
Information & Contributors
Information
Published In
1077-2626 © 2022 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.html for more information.
Publisher
IEEE Educational Activities Department
United States
Publication History
Published: 01 October 2023
Qualifiers
- Research-article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 15 Oct 2024
Other Metrics
Citations
View Options
View options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in