Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Coherent intrinsic images from photo collections

Published: 01 November 2012 Publication History

Abstract

An intrinsic image is a decomposition of a photo into an illumination layer and a reflectance layer, which enables powerful editing such as the alteration of an object's material independently of its illumination. However, decomposing a single photo is highly under-constrained and existing methods require user assistance or handle only simple scenes. In this paper, we compute intrinsic decompositions using several images of the same scene under different viewpoints and lighting conditions. We use multi-view stereo to automatically reconstruct 3D points and normals from which we derive relationships between reflectance values at different locations, across multiple views and consequently different lighting conditions. We use robust estimation to reliably identify reflectance ratios between pairs of points. From these, we infer constraints for our optimization and enforce a coherent solution across multiple views and illuminations. Our results demonstrate that this constrained optimization yields high-quality and coherent intrinsic decompositions of complex scenes. We illustrate how these decompositions can be used for image-based illumination transfer and transitions between views with consistent lighting.

References

[1]
Barrow, H., and Tenenbaum, J. 1978. Recovering intrinsic scene characteristics from images. Computer Vision Systems.
[2]
Bousseau, A., Paris, S., and Durand, F. 2009. User-assisted intrinsic images. ACM Trans. Graph. 28, 5.
[3]
Chaurasia, G., Sorkine, O., and Drettakis, G. 2011. Silhouette-aware warping for image-based rendering. Computer Graphics Forum (Proceedings of the Eurographics Symposium on Rendering) 30, 4.
[4]
Debevec, P., and et al. 2004. Estimating surface reflectance properties of a complex scene under captured natural illumination. Tech. rep., USC Institute for Creative Technologies.
[5]
Furukawa, Y., and Ponce, J. 2009. Accurate, dense, and robust multi-view stereopsis. IEEE Trans. PAMI 32, 8, 1362--1376.
[6]
Garces, E., Munoz, A., Lopez-Moreno, J., and Gutierrez, D. 2012. Intrinsic images by clustering. Computer Graphics Forum. Eurographics Symposium on rendering, EGSR '12.
[7]
Garg, R., Du, H., Seitz, S. M., and Snavely, N. 2009. The dimensionality of scene appearance. In IEEE ICCV, 1917--1924.
[8]
Grosse, R., Johnson, M. K., Adelson, E. H., and Freeman, W. T. 2009. Ground-truth dataset and baseline evaluations for intrinsic image algorithms. In IEEE ICCV.
[9]
Haber, T., Fuchs, C., Bekaert, P., Seidel, H.-P., Goesele, M., and Lensch, H. 2009. Relighting objects from image collections. In Proc. IEEE CVPR, 627--634.
[10]
Hays, J., and Efros, A. A. 2007. Scene completion using millions of photographs. ACM TOG (Proc. SIGGRAPH) 26, 3.
[11]
Hoiem, D., Efros, A. A., and Hebert, M. 2005. Automatic photo pop-up. ACM TOG (Proc. SIGGRAPH) 24, 3, 577--584.
[12]
Hoppe, H., DeRose, T., Duchamp, T., McDonald, J., and Stuetzle, W. 1992. Surface reconstruction from unorganized points. SIGGRAPH 26, 71--78.
[13]
Horn, B. K. 1986. Robot Vision, 1st ed. McGraw-Hill Higher Education.
[14]
Laffont, P.-Y., Bousseau, A., and Drettakis, G. 2012. Rich intrinsic image decomposition of outdoor scenes from multiple views. IEEE Trans. on Vis. and Comp. Graph.
[15]
Lee, K. J., Zhao, Q., Tong, X., Gong, M., Izadi, S., Uk Lee, S., Tan, P., and Lin, S. 2012. Estimation of intrinsic image sequences from image+depth video. In Proc. ECCV.
[16]
Levin, A., Lischinski, D., and Weiss, Y. 2008. A closed-form solution to natural image matting. IEEE Trans. PAMI.
[17]
Liu, X., Wan, L., Qu, Y., Wong, T.-T., Lin, S., Leung, C.-S., and Heng, P.-A. 2008. Intrinsic colorization. ACM, SIGGRAPH Asia '08, 152:1--152:9.
[18]
Matsushita, Y., Lin, S., Kang, S., and Shum, H.-Y. 2004. Estimating intrinsic images from image sequences with biased illumination. In Proc. ECCV, vol. 3022, 274--286.
[19]
Matusik, W., Loper, M., and Pfister, H. 2004. Progressively-refined reflectance functions from natural illumination. In Proc. EGSR, 299--308.
[20]
Pharr, M., and Humphreys, G. 2010. Physically Based Rendering: From Theory to Implementation, second edition. Morgan Kaufmann Publishers Inc.
[21]
Preetham, A. J., Shirley, P., and Smits, B. 1999. A practical analytic model for daylight. In SIGGRAPH, 91--100.
[22]
Roberts, D. A., 2009. Pixelstruct, an opensource tool for visualizing 3d scenes reconstructed from photographs.
[23]
Shen, L., and Yeo, C. 2011. Intrinsic image decomposition using a local and global sparse representation of reflectance. In Proc. IEEE CVPR.
[24]
Shen, L., Tan, P., and Lin, S. 2008. Intrinsic image decomposition with non-local texture cues. In Proc. IEEE CVPR.
[25]
Shen, J., Yang, X., Jia, Y., and Li, X. 2011. Intrinsic images using optimization. In Proc. IEEE CVPR.
[26]
Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: Exploring photo collections in 3d. ACM TOG (Proc. SIGGRAPH) 25, 3, 835--846.
[27]
Snavely, N., Garg, R., Seitz, S. M., and Szeliski, R. 2008. Finding paths through the world's photos. ACM TOG (Proc. SIGGRAPH) 27, 3, 11--21.
[28]
Sunkavalli, K., Matusik, W., Pfister, H., and Rusinkiewicz, S. 2007. Factored time-lapse video. ACM Transactions on Graphics (Proc. SIGGRAPH) 26, 3.
[29]
Tappen, M. F., Freeman, W. T., and Adelson, E. H. 2005. Recovering intrinsic images from a single image. IEEE Trans. PAMI 27, 9.
[30]
Troccoli, A., and Allen, P. 2008. Building illumination coherent 3d models of large-scale outdoor scenes. Int. J. Comput. Vision 78, 2--3, 261--280.
[31]
Tuite, K., Snavely, N., Hsiao, D.-y., Tabing, N., and Popovic, Z. 2011. Photocity: training experts at large-scale image acquisition through a competitive game. In Proc. SIGCHI'11, 1383--1392.
[32]
Weiss, Y. 2001. Deriving intrinsic images from image sequences. In IEEE ICCV, vol. 2, 68.
[33]
Wu, C., Agarwal, S., Curless, B., and Seitz, S. 2011. Multicore bundle adjustment. In Proc. IEEE CVPR, 3057--3064.
[34]
Yu, Y., and Malik, J. 1998. Recovering photometric properties of architectural scenes from photographs. In SIGGRAPH'98.
[35]
Yu, Y., Debevec, P., Malik, J., and Hawkins, T. 1999. Inverse global illumination: recovering reflectance models of real scenes from photographs. In SIGGRAPH '99, 215--224.
[36]
Zhao, Q., Tan, P., Dai, Q., Shen, L., Wu, E., and Lin, S. 2012. A closed-form solution to retinex with nonlocal texture constraints. IEEE Trans. PAMI 34.

Cited By

View all
  • (2023)An Overview of the Application of Machine Vision in Recognition and Localization of Fruit and Vegetable Harvesting RobotsAgriculture10.3390/agriculture1309181413:9(1814)Online publication date: 14-Sep-2023
  • (2023)IBL‐NeRF: Image‐Based Lighting Formulation of Neural Radiance FieldsComputer Graphics Forum10.1111/cgf.1492942:7Online publication date: 30-Oct-2023
  • (2023)IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00038(339-151)Online publication date: 1-Oct-2023
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 31, Issue 6
November 2012
794 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/2366145
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2012
Published in TOG Volume 31, Issue 6

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. intrinsic images
  2. photo collections

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)11
  • Downloads (Last 6 weeks)0
Reflects downloads up to 26 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2023)An Overview of the Application of Machine Vision in Recognition and Localization of Fruit and Vegetable Harvesting RobotsAgriculture10.3390/agriculture1309181413:9(1814)Online publication date: 14-Sep-2023
  • (2023)IBL‐NeRF: Image‐Based Lighting Formulation of Neural Radiance FieldsComputer Graphics Forum10.1111/cgf.1492942:7Online publication date: 30-Oct-2023
  • (2023)IntrinsicNeRF: Learning Intrinsic Neural Radiance Fields for Editable Novel View Synthesis2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00038(339-151)Online publication date: 1-Oct-2023
  • (2023)Complementary Intrinsics from Neural Radiance Fields and CNNs for Outdoor Scene Relighting2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.01593(16600-16609)Online publication date: Jun-2023
  • (2023)Seeing a Rose in Five Thousand Ways2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52729.2023.00099(962-971)Online publication date: Jun-2023
  • (2023)Reflectance edge guided networks for detail-preserving intrinsic image decompositionScience China Information Sciences10.1007/s11432-021-3481-366:2Online publication date: 5-Jan-2023
  • (2022)Unsupervised Intrinsic Image Decomposition Using Internal Self-Similarity CuesIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.312979544:12(9669-9686)Online publication date: 1-Dec-2022
  • (2022)Non-Local Representation Based Mutual Affine-Transfer Network for Photorealistic StylizationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.309594844:10_Part_2(7046-7061)Online publication date: 1-Oct-2022
  • (2022)Intrinsic Satellite Video Decomposition With Motion Target Energy ConstraintIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2022.322070460(1-13)Online publication date: 2022
  • (2022)Time-of-Day Neural Style Transfer for Architectural Photographs2022 IEEE International Conference on Computational Photography (ICCP)10.1109/ICCP54855.2022.9887763(1-12)Online publication date: 1-Aug-2022
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Share

Share

Share this Publication link

Share on social media