Image-based environment matting

Environment matting is a powerful technique for modeling the complex light-transport properties of real-world optically active elements: transparent, refractive and reflective objects. Recent research has shown how environment mattes can be computed for real objects under carefully controlled laboratory conditions. However, many objects for which environment mattes are necessary for accurate rendering cannot be placed into a calibrated lighting environment. We show in this paper that analysis of the way in which optical elements distort the appearance of their backgrounds allows the construction of environment mattes in situ without the need for specialized calibration.Specifically, given multiple images of the same element over the same background, where the element and background have relative motion, it is shown that both the background and the optical element's light-transport path can be computed.We demonstrate the technique on two different examples. In the first case, the optical element's geometry is simple, and evaluation of the realism of the output is easy. In the second, previous techniques would be difficult to apply. We show that image-based environment matting yields a realistic solution. We discuss how the stability of the solution depends on the number of images used, and how to regularize the solution where only a small number of images are available.