Abstract
Achieving convincing visual consistency between virtual objects and a real scene mainly relies on the lighting effects of virtual-real composition scenes. The problem becomes more challenging in lighting virtual objects in a single real image. Recently, scene understanding from a single image has made great progress. The estimated geometry, semantic labels and intrinsic components provide mostly coarse information, and are not accurate enough to re-render the whole scene. However, carefully integrating the estimated coarse information can lead to an estimate of the illumination parameters of the real scene. We present a novel method that uses the coarse information estimated by current scene understanding technology to estimate the parameters of a ray-based illumination model to light virtual objects in a real scene. Our key idea is to estimate the illumination via a sparse set of small 3D surfaces using normal and semantic constraints. The coarse shading image obtained by intrinsic image decomposition is considered as the irradiance of the selected small surfaces. The virtual objects are illuminated by the estimated illumination parameters. Experimental results show that our method can convincingly light virtual objects in a single real image, without any pre-recorded 3D geometry, reflectance, illumination acquisition equipment or imaging information of the image.
Similar content being viewed by others
References
Madsen C B, Nielsen M. Towards probe-less augmented reality: a position paper. In: Proceedings of the International Conference on Computer Graphics Theory and Applications, Madeira-Funchal, 2008. 255–261
Lalonde J F, Efros A, Narasimhan S. Estimating natural illumination from a single outdoor image. In: Proceedings of the IEEE International Conference on Computer Vision, Kyoto, 2009. 183–190
Liu Y L, Qin X Y, Xu S H, et al. Light source estimation of outdoor scenes for mixed reality. Vis Comput, 2009, 25: 637–646
Liu Y L, Qin X Y, Xing G Y, et al. A new approach to outdoor illumination estimation based on statistical analysis for augmented reality. J Vis Comput Animat, 2010, 21: 321–330
Chen X W, Wang K, Jin X. Single image based illumination estimation for lighting virtual object in real scene. In: Proceedings of the International Conference on Computer-Aided Design and Computer Graphics, Jinan, 2011. 450–455
Xing G Y, Liu Y L, Qin X Y, et al. On-line illumination estimation of outdoor scenes based on area selection for augmented reality. In: Proceedings of the International Conference on Computer-Aided Design and Computer Graphics, Jinan, 2011. 439–442
Liu Y L, Granier X. Online tracking of outdoor lighting variations for augmented reality with moving cameras. IEEE Trans Vis Comput Graph, 2012, 18: 573–580
Marschner S R, Greenberg D P. Inverse lighting for photography. In: Proceedings of the Color Imaging Conference: Color Science, Systems and Applications, Scottsdale, 1997. 262–265
Debevec P. Rendering synthetic objects into real scenes: bridging traditional and image-based graphics with global illumination and high dynamic range photography. In: Proceedings of the Annual Conference on Computer Graphics and Interactive Techniques, New York, 1998. 189–198
Mei X, Ling H B, Jacobs D W. Illumination recovery from image with cast shadows via sparse representation. IEEE Trans Image Process, 2011, 20: 2366–2377
Saxena A, Sun M, Ng A Y. Make3d: learning 3d scene structure from a single still image. IEEE Trans Patt Anal Mach Intell, 2009, 31: 824–840
Liu B, Gould S, Koller D. Single image depth estimation from predicted semantic labels. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, San Francisco, 2010. 1253–1260
Tappen M F, Adelson E H, Freeman W T. Estimating intrinsic component images using non-linear regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, New York, 2006. 1992–1999
Panagopoulos A, Wang C H, Samaras D, et al. Illumination estimation and cast shadow detection through a higherorder graphical model. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, 2011. 673–680
Fischler M A, Bolles R C. Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography. Commun ACM, 1981, 24: 381–395
Frahm J M, Koeser K, Grest D, et al. Markerless augmented reality with light source estimation for direct illumination. In: Proceedings of the European Conference on Visual Media Production, London, 2005. 211–220
Basri R, Jacobs D, Kemelmacher I. Photometric stereo with general, unknown lighting. Int J Comput Vis, 2007, 72: 239–257
Hoiem D, Efros A A, Hebert M. Recovering surface layout from an image. Int J Comput Vis, 2007, 75: 151–172
Gupta A, Efros A A, Hebert M. Blocks world revisited: image understanding using qualitative geometry and mechanics. In: Proceedings of the European Conference on Computer Vision, Heraklion, 2010. 482–496
Shen L, Tan P, Lin S. Intrinsic image decomposition with non-local texture cues. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Anchorage, 2008. 1–7
Bousseau A, Paris S, Durand F. User assisted intrinsic images. ACM Trans Graph, 2009, 28: 130:1–130:10
Jiang X Y, Schofield A J, Wyatt J L. Correlation-based intrinsic image extraction from a single image. In: Proceedings of the European Conference on Computer Vision, Berlin, 2010. 58–71
Barrow H, Tenenbaum J. Recovering intrinsic scene characteristics from images. In: Proceedings of the International Conference on Computer Vision Systems, Nice, 1978. 3–26
Huang P J, Gu Y T, Wu X L, et al. Time-varying clustering for local lighting and material design. Sci China Ser F-Inf Sci, 2009, 52: 445–456
Garg K, Nayar S K. Photorealistic rendering of rain streaks. ACM Trans Graph, 2006, 25: 996–1002
Wenger A, Gardner A, Tchou C, et al. Performance relighting and reflectance transformation with time-multiplexed illumination. ACM Trans Graph, 2005, 24: 756–764
Chabert C F, Einarsson P, Jones A, et al. Relighting human locomotion with flowed reflectance fields. In: Proceedings of the Eurographics Symposium on Rendering, Cyprus, 2006. 183–194
Peers P, Tamura N, Matusik W, et al. Post-production facial performance relighting using reflectance transfer. ACM Trans Graph, 26: 52:1–52:10
Jin X, Zhao M T, Chen X W, et al. Learning artistic lighting template from portrait photographs. In: Proceedings of the European Conference on Computer Vision, Heraklion, 2010. 101–114
Chen X W, Chen M M, Jin X, et al. Face illumination transfer through edge-preserving filters. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, 2011. 281–287
Chen X W, Jin X, Zhao Q P, et al. Artistic illumination transfer for portraits. Comput Graph Forum, 2012, 31: 1425–1434
Supan P, Stuppacher I, Haller M. Image based shadowing in real-time augmented reality. Int J Virtual Real, 2006, 5: 1–7
Pilet J, Geiger A, Lagger P, et al. An all-in-one solution to geometric and photometric calibration. In: Proceedings of the IEEE and ACM Symposium on Mixed and Augmented Reality, Santa Barbara, 2006. 69–78
Jensen T, Andersen M S, Madsen C B. Real-time image based lighting for outdoor augmented reality under dynamically changing illumination conditions. In: Proceedings of the International Conference on Computer Graphics Theory and Applications, Setúbal, 2006. 364–371
Haber T, Fuchs C, Bekaert P, et al. Relighting objects from image collections. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Miami, 2009. 627–634
Karsch K, Hedau V, Forsyth D, et al. Rendering synthetic objects into legacy photographs. ACM Trans Graph, 2011, 30: 157:1–157:12
Phong B T. Illumination for computer generated pictures. Commun ACM, 1975, 18: 311–317
Moré J J. The Levenberg-Marquardt algorithm: implementation and theory. Numer Anal, 1977, 630: 105–116
Lalonde J F, Efros A A, Narasimhan S G. Webcam clip art: appearance and illuminant transfer from time-lapse sequences. ACM Trans Graph, 2009, 28: 131:1–131:10
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Chen, X., Jin, X. & Wang, K. Lighting virtual objects in a single image via coarse scene understanding. Sci. China Inf. Sci. 57, 1–14 (2014). https://doi.org/10.1007/s11432-013-4936-0
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s11432-013-4936-0