research-article

Interactive Virtual Relighting of Real Scenes

Authors:

Céline Loscos,

George Drettakis,

Luc RobertAuthors Info & Claims

IEEE Transactions on Visualization and Computer Graphics, Volume 6, Issue 4

Pages 289 - 305

https://doi.org/10.1109/2945.895874

Published: 01 October 2000 Publication History

Abstract

Computer augmented reality (CAR) is a rapidly emerging field which enables users to mix real and virtual worlds. Our goal is to provide interactive tools to perform common illumination, i.e., light interactions between real and virtual objects, including shadows and relighting (real and virtual light source modification). In particular, we concentrate on virtually modifying real light source intensities and inserting virtual lights and objects into a real scene; such changes can be very useful for virtual lighting design and prototyping. To achieve this, we present a three-step method. We first reconstruct a simplified representation of real scene geometry using semiautomatic vision-based techniques. With the simplified geometry, and by adapting recent hierarchical radiosity algorithms, we construct an approximation of real scene light exchanges. We next perform a preprocessing step, based on the radiosity system, to create unoccluded illumination textures. These replace the original scene textures which contained real light effects such as shadows from real lights. This texture is then modulated by a ratio of the radiosity (which can be changed) over a display factor which corresponds to the radiosity for which occlusion has been ignored. Since our goal is to achieve a convincing relighting effect, rather than an accurate solution, we present a heuristic correction process which results in visually plausible renderings. Finally, we perform an interactive process to compute new illumination with modified real and virtual light intensities. Our results show that we are able to virtually relight real scenes interactively, including modifications and additions of virtual light sources and objects.

References

[1]

http://www.inria.fr/robotvis/personnel/sbougnou/TotalCalib/

[2]

R.T. Azuma, “A Survey of Augmented Reality,” Presence: Teleoperators and Virtual Environments 6, pp. 355-385, Aug. 1997, earlier version in “Course Notes #9: Developing Advanced Virtual Reality Applications,” ACM SIGGRAPH, pp. 20-1-20-38, 1995.

Digital Library

[3]

M.F. Cohen S.E. Chen J.R. Wallace and D.P. Greenberg, “A Progressive Refinement Approach to Fast Radiosity Image Generation,” Computer Graphics (SIGGRAPH '88 Proc.), J. Dill, ed., vol. 22, pp. 75-84, Aug. 1988.

Digital Library

[4]

P.E. Debevec, “Rendering Synthetic Objects into Real Scenes: Bridging Traditional and Image-Based Graphics with Global Illumination and High Dynamic Range Photography,” SIGGRAPH '98 Conf. Proc., M. Cohen, ed., pp. 189-198, 1998.

Digital Library

[5]

P.E. Debevec G. Borshukov and Y. Yu, “Efficient View-Dependent Image-Based Rendering with Projective Texture-Mapping,” Proc. Rendering Techniques '98 (Ninth EG Workshop Rendering), G. Drettakis and N. Max, eds., June 1998.

[6]

P.E. Debevec and J. Malik, “Recovering High Dynamic Range Radiance Maps from Photographs,” SIGGRAPH '97 Conf. Proc., T. Whitted, ed., pp. 369-378, Aug. 1997.

Digital Library

[7]

P.E. Debevec C.J. Taylor and J. Malik, “Modeling and Rendering Architecture from Photographs: A Hybrid Geometry- and Image-Based Approach,” SIGGRAPH '96 Conf. Proc., H. Rushmeier, ed., pp. 11-20, Aug. 1996.

Digital Library

[8]

G. Drettakis L. Robert and S. Bougnoux, “Interactive Common Illumination for Computer Augmented Reality,” Proc. Rendering Techniques '98 (Eighth EG Workshop Rendering), J. Dorsey and P. Slusallek, eds., pp. 45-56, June 1997.

Digital Library

[9]

G. Drettakis and F. Sillion, “Interactive Update of Global Illumination Using a Line-Space Hierarchy,” SIGGRAPH '97 Conf. Proc., T. Whitted, ed., pp. 57-64, Aug. 1997.

Digital Library

[10]

M.D. Fairchild, Color Appearance Models. Addison-Wesley, 1998.

[11]

O. Faugeras S. Laveau L. Robert G. Csurka C. Zeller C. Gauclin and I. Zoghlami, “3-D Reconstruction of Urban Scenes from Image Sequences,” CVGIP: Image Understanding, 1997.

Digital Library

[12]

A. Fournier A.S. Gunawan and C. Romanzin, “Common Illumination between Real and Computer Generated Scenes,” Proc. Graphics Interface '93, pp. 254-262, May 1993.

[13]

S, Gibson and R.J. Hubbold, “Efficient Hierarchical Refinement and Clustering for Radiosity in Complex Environments,” Computer Graphics Forum, vol. 15, no. 5, pp. 297-310, 1996.

[14]

E. Haines and J. Wallace, “Shaft Culling for Efficient Ray-Traced Radiosity,” Photorealistic Rendering Techniques in Computer Graphics (Proc. Second EG Workshop Rendering), P. Brunet and F.W. Jansen, eds., pp. 122-138, 1991.

[15]

P. Hanrahan D. Salzman and L. Aupperle, “A Rapid Hierarchical Radiosity Algorithm,” Computer Graphics (SIGGRAPH '91 Proc.), T.W. Sederberg, ed., vol. 25, pp. 197-206, July 1991.

Digital Library

[16]

P. Jancène F. Neyret X. Provot J.-P. Tarel J.-M. Vézien C. Meilhac and A. Vérroust, “Res: Computing the Interactions between Real and Virtual Objects in Video Sequences,” Proc. Second IEEE Workshop Networked Realities, pp. 27-40, Oct. 1995. http://www-rocq.inria.fr/syntim/textes/nr95-eng.html.

[17]

D. Lischinski F. Tampieri and D.P. Greenberg, “Combining Hierarchical Radiosity and Discontinuity Meshing,” Computer Graphics Proc., Ann. Conf. Series, pp. 199-208, 1993.

Digital Library

[18]

C. Loscos M.-C. Frasson G. Drettakis B. Walter X. Granier and P. Poulin, “Interactive Virtual Relighting and Remodeling of Real Scenes,” Proc. Rendering Techniques '99 (10th EG Workshop Rendering), D. Lischinski and G. Ward-Larson, eds., June 1999.

Digital Library

[19]

E. Nakamae K. Harada T. Ishizaki and T. Nishita, “A Montage Method: The Overlaying of the Computer Generated Images onto a Background Photograph,” Computer Graphics (SIGGRAPH '86 Proc.), D.C. Evans and R.J. Athay, eds., vol. 20, pp. 207-214, Aug. 1986.

Digital Library

[20]

P. Poulin M. Ouimet and M.-C. Frasson, “Interactively Modeling with Photogrammetry,” Proc. Rendering Techniques '98 (Ninth EG Workshop Rendering), G. Drettakis and N. Max, eds., pp. 93-104, June 1998.

[21]

L. Robert, “Camera Calibration without Feature Extraction,” Computer Vision, Graphics, and Image Processing, vol. 63, no. 2, pp. 314-325, Mar. 1995, also INRIA Technical Report 2204.

Digital Library

[22]

I. Sato Y. Sato and K. Ikeuchi, “Acquiring a Radiance Distribution to Superimpose Virtual Objects onto a Real Scene,” IEEE Trans. Visualization and Computer Graphics, vol. 5, no. 1, pp. 1-12, Jan.-Mar. 1999.

Digital Library

[23]

F.X. Sillion, “A Unified Hierarchical Algorithm for Global Ilumination with Scattering Volumes and Object Clusters,” IEEE Trans. Visualization and Computer Graphics, vol. 1, no. 3, pp. 240-254, Sept. 1995.

Digital Library

[24]

F.X. Sillion G. Drettakis and C. Soler, “A Clustering Algorithm for Radiance Calculation in General Environments,” Proc. Rendering Techniques '95 (Fifth EG Workshop Rendering), P.M. Hanrahan and W. Purgathofer, eds., pp. 196-205, June 1995.

[25]

F.X. Sillion and C. Puech, Radiosity and Global Illumination. San Francisco: Morgan Kaufmann, 1994.

Digital Library

[26]

B. Smits J. Arvo and D. Greenberg, “A Clustering Algorithm for Radiosity in Complex Rnvironments,” SIGGRAPH '94 Conference Proc., A. Glassner, ed., pp. 435-442, July 1994.

Digital Library

[27]

C. Soler and F. Sillion, “Automatic Calculation of Soft Shadow Textures for Fast, High-Quality Radiosity,” Proc. Rendering Techniques '98 (Ninthth EG Workshop Rendering), G. Drettakis and N. Max, eds., June 1998.

[28]

A. State G. Hirota D.T. Chen B. Garrett and M. Livingston, “Superior Augmented Reality Registration by Integrating Landmark Tracking and Magnetic Tracking,” SIGGRAPH '96 Conf. Proc., H. Rushmeier, ed., pp. 429-438, Aug. 1996.

Digital Library

[29]

G.J. Ward, “The RADIANCE Lighting Simulation and Rendering System,” SIGGRAPH '94 Conf. Proc., A. Glassner, ed., pp. 459-472, July 1994.

Digital Library

[30]

Y. Yu P.E. Debevec J. Malik and T. Hawkins, “Inverse Global Illumination: Recovering Reflectance Models of Real Scenes from Photographs,” SIGGRAPH '99 Conf. Proc., A. Rockwood, ed., 1999.

Digital Library

[31]

Y. Yu and J. Malik, “Recovering Photometric Properties of Architectural Scenes from Photographs,” SIGGRAPH '98 Conf. Proc., M. Cohen, ed., pp. 207-218, July 1998.

Digital Library

[32]

Z. Zhang R. Deriche O. Faugeras and Q.-T. Luong, “A Robust Technique for Matching Two Uncalibrated Images through the Recovery of the Unknown Epipolar Geometry,” Artificial Intelligence J., vol. 78, pp. 87-119, Oct. 1995.

Digital Library

[33]

I. Zoghlami O. Faugeras and R. Deriche, “Using Geometric Corners to Build a 2D Mosaic from a Set of Images” Proc. Int'l Conf. Computer Vision and Pattern Recognition, June 1997.

Digital Library

Cited By

Alhakamy ATuceryan M(2020)Real-time Illumination and Visual Coherence for Photorealistic Augmented/Mixed RealityACM Computing Surveys10.1145/338649653:3(1-34)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3386496
Kolivand HSunar M(2014)Covering photo-realistic properties of outdoor components with the effects of sky color in mixed realityMultimedia Tools and Applications10.1007/s11042-013-1494-972:3(2143-2162)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.1007/s11042-013-1494-9
Okura FBabaguchi NAizawa KSmith JSatoh SPlagemann THua XYan R(2012)Spacetime freeview generation using image-based rendering, relighting, and augmented telepresenceProceedings of the 20th ACM international conference on Multimedia10.1145/2393347.2396513(1437-1440)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2393347.2396513
Show More Cited By

Index Terms

Interactive Virtual Relighting of Real Scenes
1. Computing methodologies
  1. Computer graphics

Recommendations

Interactive relighting with dynamic BRDFs

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance ...
Interactive relighting with dynamic BRDFs
SIGGRAPH '07: ACM SIGGRAPH 2007 papers

We present a technique for interactive relighting in which source radiance, viewing direction, and BRDFs can all be changed on the fly. In handling dynamic BRDFs, our method efficiently accounts for the effects of BRDF modification on the reflectance ...
Textures and radiosity: controlling emission and reflection with texture maps
SIGGRAPH '94: Proceedings of the 21st annual conference on Computer graphics and interactive techniques

In this paper we discuss the efficient and accurate incorporation of texture maps into a hierarchical Galerkin radiosity algorithm. This extension of the standard algorithm allows the use of textures to describe complex reflectance and emittance ...

Comments

Information & Contributors

Information

Published In

cover image IEEE Transactions on Visualization and Computer Graphics

IEEE Transactions on Visualization and Computer Graphics Volume 6, Issue 4

October 2000

99 pages

ISSN:1077-2626

Issue’s Table of Contents

Copyright © Copyright © 2000 IEEE. All Rights Reserved.

Publisher

IEEE Educational Activities Department

United States

Publication History

Published: 01 October 2000

Author Tags

Qualifiers

Research-article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

18
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Alhakamy ATuceryan M(2020)Real-time Illumination and Visual Coherence for Photorealistic Augmented/Mixed RealityACM Computing Surveys10.1145/338649653:3(1-34)Online publication date: 28-May-2020
https://dl.acm.org/doi/10.1145/3386496
Kolivand HSunar M(2014)Covering photo-realistic properties of outdoor components with the effects of sky color in mixed realityMultimedia Tools and Applications10.1007/s11042-013-1494-972:3(2143-2162)Online publication date: 1-Oct-2014
https://dl.acm.org/doi/10.1007/s11042-013-1494-9
Okura FBabaguchi NAizawa KSmith JSatoh SPlagemann THua XYan R(2012)Spacetime freeview generation using image-based rendering, relighting, and augmented telepresenceProceedings of the 20th ACM international conference on Multimedia10.1145/2393347.2396513(1437-1440)Online publication date: 29-Oct-2012
https://dl.acm.org/doi/10.1145/2393347.2396513
Kopf JNeubert BChen BCohen MCohen-Or DDeussen OUyttendaele MLischinski D(2008)Deep photoACM SIGGRAPH Asia 2008 papers10.1145/1457515.1409069(1-10)Online publication date: 10-Dec-2008
https://dl.acm.org/doi/10.1145/1457515.1409069
Drettakis GDigalakis VPotamianos ATurk MPieraccini RIvanov Y(2008)Audiovisual 3d rendering as a tool for multimodal interfacesProceedings of the 10th international conference on Multimodal interfaces10.1145/1452392.1452394(203-204)Online publication date: 20-Oct-2008
https://dl.acm.org/doi/10.1145/1452392.1452394
Kopf JNeubert BChen BCohen MCohen-Or DDeussen OUyttendaele MLischinski D(2008)Deep photoACM Transactions on Graphics10.1145/1409060.140906927:5(1-10)Online publication date: 1-Dec-2008
https://dl.acm.org/doi/10.1145/1409060.1409069
Hwang YKim JHong H(2007)Using camera calibration and radiosity on GPU for interactive common illuminationProceedings of the 13th International conference on Multimedia Modeling - Volume Part II10.1007/978-3-540-69429-8_29(289-295)Online publication date: 9-Jan-2007
https://dl.acm.org/doi/10.1007/978-3-540-69429-8_29
Grasset RBoissieux LGascuel JSchmalstieg D(2005)Interactive mediated realityProceedings of the Sixth Australasian conference on User interface - Volume 4010.5555/1082243.1082247(21-29)Online publication date: 30-Jan-2005
https://dl.acm.org/doi/10.5555/1082243.1082247
Bimber ORaskar R(2005)Modern approaches to augmented realityACM SIGGRAPH 2005 Courses10.1145/1198555.1198711(1-es)Online publication date: 31-Jul-2005
https://dl.acm.org/doi/10.1145/1198555.1198711
Park HPark JLee S(2005)Image-Based relighting of moving objects with specular reflectionProceedings of the 13th international conference on Image Analysis and Processing10.1007/11553595_64(519-526)Online publication date: 6-Sep-2005
https://dl.acm.org/doi/10.1007/11553595_64
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents