research-article

Image-based rendering in the gradient domain

Authors:

Fabian Langguth,

Daniel Scharstein,

Richard Szeliski,

Michael GoeseleAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 32, Issue 6

Article No.: 199, Pages 1 - 9

https://doi.org/10.1145/2508363.2508369

Published: 01 November 2013 Publication History

Abstract

We propose a novel image-based rendering algorithm for handling complex scenes that may include reflective surfaces. Our key contribution lies in treating the problem in the gradient domain. We use a standard technique to estimate scene depth, but assign depths to image gradients rather than pixels. A novel view is obtained by rendering the horizontal and vertical gradients, from which the final result is reconstructed through Poisson integration using an approximate solution as a data term. Our algorithm is able to handle general scenes including reflections and similar effects without explicitly separating the scene into reflective and transmissive parts, as required by previous work. Our prototype renderer is fully implemented on the GPU and runs in real time on commodity hardware.

References

[1]

Beery, E., and Yeredor, A. 2008. Blind separation of superimposed shifted images using parameterized joint diagonalization. IEEE Transactions on Image Processing 17, 3, 340--353.

Digital Library

[2]

Bergen, J. R., Burt, P. J., Hingorani, R., and Peleg, S. 1992. A three-frame algorithm for estimating two-component image motion. IEEE Transactions on Pattern Analysis and Machine Intelligence 14, 9, 886--896.

Digital Library

[3]

Bhat, D. N., and Nayar, S. K. 1998. Stereo and specular reflection. International Journal of Computer Vision 26, 2, 91--106.

Digital Library

[4]

Buehler, C., Bosse, M., McMillan, L., Gortler, S. J., and Cohen, M. F. 2001. Unstructured Lumigraph rendering. Proc. SIGGRAPH 2001, 425--432.

Digital Library

[5]

Carceroni, R. L., and Kutulakos, K. N. 2002. Multi-view scene capture by surfel sampling: From video streams to nonrigid 3D motion, shape and reflectance. International Journal of Computer Vision 49, 2/3, 175--214.

Digital Library

[6]

Chaurasia, G., Duchene, S., Sorkine-Hornung, O., and Drettakis, G. 2013. Depth synthesis and local warps for plausible image-based navigation. ACM Transactions on Graphics 32, 3, Article no. 30.

Digital Library

[7]

Chen, S., and Williams, L. 1993. View interpolation for image synthesis. Proc. SIGGRAPH '93, 279--288.

Digital Library

[8]

Criminisi, A., Kang, S. B., Swaminathan, R., Szeliski, R., and Anandan, P. 2005. Extracting layers and analyzing their specular properties using epipolar-plane-image analysis. Computer Vision and Image Understanding 97, 1, 51--85.

Digital Library

[9]

Debevec, P. E., Taylor, C. J., and Malik, J. 1996. Modeling and rendering architecture from photographs: A hybrid geometry-and image-based approach. Proc. SIGGRAPH '96, 11--20.

Digital Library

[10]

Diamant, Y., and Schechner, Y. Y. 2008. Overcoming visual reverberations. Proc. Computer Vision and Pattern Recognition (CVPR'08), 1--8.

[11]

Eisemann, M., De Decker, B., Magnor, M., Bekaert, P., de Aguiar, E., Ahmed, N., Theobalt, C., and Sellent, A. 2008. Floating textures. Computer Graphics Forum (Proc. Eurographics 2008) 27, 2, 409--418.

[12]

Goesele, M., Ackermann, J., Fuhrmann, S., Haubold, C., Klowsky, R., Steedly, D., and Szeliski, R. 2010. Ambient point clouds for view interpolation. ACM Transactions on Graphics (Proc. SIGGRAPH 2010) 29, 4, Article no. 95.

Digital Library

[13]

Gortler, S. J., Grzeszczuk, R., Szeliski, R., and Cohen, M. F. 1996. The Lumigraph. Proc. SIGGRAPH '96, 43--54.

Digital Library

[14]

Hirschmüller, H. 2008. Stereo processing by semiglobal matching and mutual information. IEEE Transactions on Pattern Analysis and Machine Intelligence 30, 2, 328--341.

Digital Library

[15]

Irani, M., Rousso, B., and Peleg, S. 1994. Computing occluding and transparent motions. International Journal of Computer Vision 12, 1, 5--16.

Digital Library

[16]

Ju, S. X., Black, M. J., and Jepson, A. D. 1996. Skin and bones: Multi-layer, locally affine, optical flow and regularization with transparency. Proc. Computer Vision and Pattern Recognition (CVPR '96), 307--314.

Digital Library

[17]

Kolmogorov, V., and Zabih, R. 2002. Multi-camera scene reconstruction via graph cuts. Proc. European Conference on Computer Vision 2002 (ECCV 2002), 82--96.

Digital Library

[18]

Levin, A., Zomet, A., and Weiss, Y. 2004. Separating reflections from a single image using local features. Proc. Computer Vision and Pattern Recognition 2004 (CVPR 2004) 1, 306--313.

[19]

Levoy, M., and Hanrahan, P. 1996. Light field rendering. Proc. SIGGRAPH '96, 31--42.

Digital Library

[20]

Linz, C., Lipski, C., and Magnor, M. 2010. Multi-image interpolation based on graph-cuts and symmetric optical flow. Proc. Vision, Modeling and Visualization 2010 (VMV 2010), 115--122.

[21]

Loop, C., and Zhang, Z. 1999. Computing rectifying homographies for stereo vision. Proc. Computer Vision and Pattern Recognition (CVPR '99), 125--131.

[22]

Mahajan, D., Huang, F.-C., Matusik, W., Ramamoorthi, R., and Belhumeur, P. 2009. Moving gradients: a path-based method for plausible image interpolation. ACM Transactions on Graphics (Proc. SIGGRAPH 2009) 28, 3, Article no. 42.

Digital Library

[23]

Pérez, P., Gangnet, M., and Blake, A. 2003. Poisson image editing. ACM Transactions on Graphics (Proc. SIGGRAPH 2003) 22, 3, 313--318.

Digital Library

[24]

Popescu, V., Mei, C., Dauble, J., and Sacks, E. 2006. Reflected-scene impostors for realistic reflections at interactive rates. Computer Graphics Forum (Proc. Eurographics 2006) 25, 3, 313--322.

[25]

Schechner, Y. Y., Shamir, J., and Kiryati, N. 1999. Polarization-based decorrelation of transparent layers: The inclination angle of an invisible surface. Proc. International Conference on Computer Vision (ICCV '99), 814--819.

Digital Library

[26]

Schechner, Y. Y., Kiryati, N., and Shamir, J. 2000. Blind recovery of transparent and semireflected scenes. Proc. Computer Vision and Pattern Recognition (CVPR 2000), 38--43.

[27]

Shade, J., Gortler, S., He, L., and Szeliski, R. 1998. Layered depth images. Proc. SIGGRAPH '98, 231--242.

Digital Library

[28]

Shewchuk, J. R. 1994. An introduction to the conjugate gradient method without the agonizing pain. Tech. rep., Carnegie Mellon University.

Digital Library

[29]

Shizawa, M., and Mase, K. 1991. A unified computational theory of motion transparency and motion boundaries based on eigenenergy analysis. Proc. Computer Vision and Pattern Recognition (CVPR '91), 289--295.

[30]

Shum, H.-Y., Chan, S.-C., and Kang, S. B. 2007. Image-Based Rendering. Springer, New York, NY.

Digital Library

[31]

Sinha, S. N., Steedly, D., and Szeliski, R. 2009. Piecewise planar stereo for image-based rendering. Proc. International Conference on Computer Vision (ICCV 2009), 1881--1888.

[32]

Sinha, S. N., Kopf, J., Goesele, M., Scharstein, D., and Szeliski, R. 2012. Image-based rendering for scenes with reflections. ACM Transactions on Graphics (Proc. SIGGRAPH 2012) 31, 4, Article no. 100.

Digital Library

[33]

Snavely, N., Seitz, S. M., and Szeliski, R. 2006. Photo tourism: exploring photo collections in 3D. ACM Transactions on Graphics (Proc. SIGGRAPH 2006) 25, 3, 835--846.

Digital Library

[34]

Szeliski, R., Avidan, S., and Anandan, P. 2000. Layer extraction from multiple images containing reflections and transparency. Proc. Computer Vision and Pattern Recognition (CVPR 2000), 246--253.

[35]

Tsin, Y., Kang, S. B., and Szeliski, R. 2006. Stereo matching with linear superposition of layers. IEEE Transactions on Pattern Analysis and Machine Intelligence 28, 2, 290--301.

Digital Library

[36]

Zitnick, C. L., Kang, S. B., Uyttendaele, M., Winder, S., and Szeliski, R. 2004. High-quality video view interpolation using a layered representation. ACM Transactions on Graphics (Proc. SIGGRAPH 2004) 23, 3, 600--608.

Digital Library

Cited By

Zhang ZXiao JLiao LWang MCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)RefScale: Multi-temporal Assisted Image Rescaling in Repetitive Observation ScenariosProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681254(9866-9874)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681254
Lazri ZYeol Lee DSu G(2024)A Framework for Single-View Multi-Plane Image Inpainting2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR62202.2024.00092(536-541)Online publication date: 7-Aug-2024
https://doi.org/10.1109/MIPR62202.2024.00092
Wang LFrisvad JBo Jensen MBigdeli S(2024)StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Models2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00737(7416-7425)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00737
Show More Cited By

Index Terms

Image-based rendering in the gradient domain
1. Computing methodologies
  1. Computer graphics
    1. Image manipulation
    2. Rendering

Recommendations

Image-based rendering of diffuse, specular and glossy surfaces from a single image
SIGGRAPH '01: Proceedings of the 28th annual conference on Computer graphics and interactive techniques

In this paper, we present a new method to recover an approximation of the bidirectional reflectance distribution function (BRDF) of the surfaces present in a real scene. This is done from a single photograph and a 3D geometric model of the scene. The ...
An introduction to image-based rendering
Integrated image and graphics technologies

In this chapter, we review the techniques for image-based rendering. Unlike traditional 3D computer graphics in which 3D geometry of the scene is known, image-based rendering (IBR) techniques render novel views directly from input images. IBR techniques ...
Image-based rendering for scenes with reflections

We present a system for image-based modeling and rendering of real-world scenes containing reflective and glossy surfaces. Previous approaches to image-based rendering assume that the scene can be approximated by 3D proxies that enable view ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 32, Issue 6

November 2013

671 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2508363

Issue’s Table of Contents

Copyright © 2013 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 November 2013

Published in TOG Volume 32, Issue 6

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

54
Total Citations
View Citations
900
Total Downloads

Downloads (Last 12 months)13
Downloads (Last 6 weeks)1

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhang ZXiao JLiao LWang MCai JKankanhalli MPrabhakaran BBoll SSubramanian RZheng LSingh VCesar PXie LXu D(2024)RefScale: Multi-temporal Assisted Image Rescaling in Repetitive Observation ScenariosProceedings of the 32nd ACM International Conference on Multimedia10.1145/3664647.3681254(9866-9874)Online publication date: 28-Oct-2024
https://dl.acm.org/doi/10.1145/3664647.3681254
Lazri ZYeol Lee DSu G(2024)A Framework for Single-View Multi-Plane Image Inpainting2024 IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR)10.1109/MIPR62202.2024.00092(536-541)Online publication date: 7-Aug-2024
https://doi.org/10.1109/MIPR62202.2024.00092
Wang LFrisvad JBo Jensen MBigdeli S(2024)StereoDiffusion: Training-Free Stereo Image Generation Using Latent Diffusion Models2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW)10.1109/CVPRW63382.2024.00737(7416-7425)Online publication date: 17-Jun-2024
https://doi.org/10.1109/CVPRW63382.2024.00737
Pack CSoh LLorang E(2024)Perceptual cue-guided adaptive image downscaling for enhanced semantic segmentation on large document imagesInternational Journal on Document Analysis and Recognition10.1007/s10032-023-00454-727:2(159-175)Online publication date: 1-Jun-2024
https://dl.acm.org/doi/10.1007/s10032-023-00454-7
Zhou KZhong JShin SLu KYang YMarkham ATrigoni NOh ANaumann TGloberson ASaenko KHardt MLevine S(2023)DynPointProceedings of the 37th International Conference on Neural Information Processing Systems10.5555/3666122.3669167(69532-69545)Online publication date: 10-Dec-2023
https://dl.acm.org/doi/10.5555/3666122.3669167
Song SZhong FWang TQin XTu C(2023)Guided Linear UpsamplingACM Transactions on Graphics10.1145/359245342:4(1-12)Online publication date: 26-Jul-2023
https://dl.acm.org/doi/10.1145/3592453
Minelli GPoggi MSalti S(2023)Depth Self-Supervision for Single Image Novel View Synthesis2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)10.1109/IROS55552.2023.10342058(5836-5843)Online publication date: 1-Oct-2023
https://doi.org/10.1109/IROS55552.2023.10342058
Wang CWang YManocha D(2023)LoLep: Single-View View Synthesis with Locally-Learned Planes and Self-Attention Occlusion Inference2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00995(10807-10817)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00995
Cai SChan EPeng SShahbazi MObukhov AVan Gool LWetzstein G(2023)DiffDreamer: Towards Consistent Unsupervised Single-view Scene Extrapolation with Conditional Diffusion Models2023 IEEE/CVF International Conference on Computer Vision (ICCV)10.1109/ICCV51070.2023.00204(2139-2150)Online publication date: 1-Oct-2023
https://doi.org/10.1109/ICCV51070.2023.00204
Kopanas GLeimkühler TRainer GJambon CDrettakis G(2022)Neural Point Catacaustics for Novel-View Synthesis of ReflectionsACM Transactions on Graphics10.1145/3550454.355549741:6(1-15)Online publication date: 30-Nov-2022
https://dl.acm.org/doi/10.1145/3550454.3555497
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents