Article

Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing

Authors:

Ashok Veeraraghavan,

Jack TumblinAuthors Info & Claims

SIGGRAPH '07: ACM SIGGRAPH 2007 papers

Pages 69 - es

https://doi.org/10.1145/1275808.1276463

Published: 29 July 2007 Publication History

Abstract

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera image without any additional refractive elements as required by previous light field cameras. The patterned mask attenuates light rays inside the camera instead of bending them, and the attenuation recoverably encodes the rays on the 2D sensor. Our mask-equipped camera focuses just as a traditional camera to capture conventional 2D photos at full sensor resolution, but the raw pixel values also hold a modulated 4D light field. The light field can be recovered by rearranging the tiles of the 2D Fourier transform of sensor values into 4D planes, and computing the inverse Fourier transform. In addition, one can also recover the full resolution image information for the in-focus parts of the scene.

We also show how a broadband mask placed at the lens enables us to compute refocused images at full sensor resolution for layered Lambertian scenes. This partial encoding of 4D ray-space data enables editing of image contents by depth, yet does not require computational recovery of the complete 4D light field.

Supplementary Material

JPG File (pps069.jpg)

Download
15.58 KB

MP4 File (pps069.mp4)

Download
55.87 MB

References

[1]

Accorsi, R., Gasparini, F., and Lanza, R. C. 2001. Optimal coded aperture patterns for improved SNR in nuclear medicine imaging. Nuclear Instruments and Methods in Physics Research A 474 (Dec.), 273--284.

[2]

Adelson, T., and Wang, J. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Machine Intell. 14, 99--106.

Digital Library

[3]

Agarwala, A., Dontcheva, M., Agrawala, M., Drucker, S., Colburn, A., Curless, B., Salesin, D., and Cohen, M. 2004. Interactive digital photomontage. ACM Trans. Graph. 23, 3, 294--302.

Digital Library

[4]

Boykov, Y., Veksler, O., and Zabih, R. 2001. Fast approximate energy minimization using graph cuts. IEEE Trans. Pattern Anal. Machine Intell. 23, 1222--1239.

Digital Library

[5]

Chaudhuri, S., and Rajagopalan, A. 1999. Depth from Defocus: A Real Aperture Imaging Approach. Springer.

[6]

Dowski, E. R., and Cathey, W. 1995. Extended depth of field through wavefront coding. Appl. Optics 34, 11 (Apr.), 1859--1866.

[7]

Dowski, E. R., and Johnson, G. E. 1999. Wavefront coding: A modern method of achieving high performance and/or low cost imaging systems. In SPIE Annual Meeting.

[8]

Farid, H., and Simoncelli, E. 1998. Range estimation by optical differentiation. J. Opt. Soc. of America A 15, 1, 1777--1786.

[9]

Fergus, R., Torralba, A., and Freeman, W. 2006. Random lens imaging. Tech. rep., MIT.

[10]

Fessenden, R. 1908. Wireless telephony. Trans. American Institute of Electrical Engineers 27, 553--629.

[11]

Field, D. 1994. What is the goal of sensory coding? Neural Comput. 6, 559--601.

Digital Library

[12]

Georgiev, T., Zheng, C., Nayar, S., Curless, B., Salasin, D., and Intwala, C. 2006. Spatio-angular resolution trade-offs in integral photography. In Eurographics Symposium on Rendering, 263--272.

[13]

Gortler, S., Grzeszczuk, R., Szeliski, R., and Cohen, M. 1996. The lumigraph. In SIGGRAPH, 43--54.

Digital Library

[14]

Gottesman, S. R., and Fenimore, E. E. 1989. New family of binary arrays for coded aperture imaging. Appl. Optics 28, 20 (Oct), 4344--4352.

[15]

Haeberli, P., 1994. A multifocus method for controlling depth of field. GraficaObscura.

[16]

Hiura, S., and Matsuyama, T. 1998. Depth measurement by the multi-focus camera. In Proc. Conf. Computer Vision and Pattern Recognition, 953--961.

Digital Library

[17]

Imatest. Image quality evaluation software. http://www.imatest.com/.

[18]

Isaksen, A., McMillan, L., and Gortler, S. 2000. Dynamically reparameterized light fields. In SIGGRAPH, 297--306.

Digital Library

[19]

Ives, H. 1928. Camera for making parallax panoramagrams. J. Opt. Soc. Amer. 17, 435--439.

[20]

Javidi, B., and Okano, F., Eds. 2002. Three-Dimensional Television, Video and Display Technologies. Springer-Verlag.

Digital Library

[21]

Johnson, G. E., Dowski, E. R., and Cathey, W. T. 2000. Passive ranging through wave-front coding: Information and application. Applied Optics 39, 1700--1710.

[22]

Levoy, M., and Hanrahan, P. 1996. Light field rendering. In SIGGRAPH 96, 31--42.

Digital Library

[23]

Levoy, M., Chen, B., Vaish, V., Horowitz, M., McDowall, M., and Bolas, M. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 825--834.

Digital Library

[24]

Lippmann, G. 1908. Epreuves reversible donnant la sensation du relief. J. Phys 7, 821--825.

[25]

Lucy, L. 1974. An iterative technique for the rectification of observed distributions. J. Astronomy 79, 745--754.

[26]

Martnez-Corral, M., Javidi, B., Martnez-Cuenca, R., and Saavedra, G. 2004. Integral imaging with improved depth of field by use of amplitude-modulated microlens arrays. Applied Optics 43, 5806--5813.

[27]

Nayar, S., and Mitsunaga, T. 2000. High dynamic range imaging: spatially varying pixel exposures. In Proc. Conf. Computer Vision and Pattern Recognition, vol. 1, 472--479.

[28]

Nayar, S. K., Branzoi, V., and Boult, T. E. 2006. Programmable imaging: Towards a flexible camera. Int'l J. Computer Vision 70, 1, 7--22.

Digital Library

[29]

Ng, R., Levoy, M., Brdif, M., Duval, G., Horowitz, M., and Hanrahan, P. 2005. Light field photography with a hand-held plenoptic camera. Tech. rep., Stanford Univ.

[30]

Ng, R. 2005. Fourier slice photography. ACM Trans. Graph. 24, 735--744.

Digital Library

[31]

Okano, F., Hoshino, H., and Yuyama, A. 1997. Real-time pickup method for a three-dimensional image based on integral photography. Applied Optics 36, 15981603.

[32]

Okano, F., Arai, J., Hoshino, H., and Yuyama, I. 1999. Three dimensional video system based on integral photography. Optical Engineering 38, 1072--1077.

[33]

Oppenheim, A. V., Schafer, R. W., and Buck, J. R. 1999. Discrete-Time Signal Processing. Prentice-Hall.

Digital Library

[34]

Raskar, R., Agrawal, A., and Tumblin, J. 2006. Coded exposure photography: motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795--804.

Digital Library

[35]

Richardson, W. 1972. Bayesian-based iterative method of image restoration. J. Opt. Soc. of Am. 62, 1, 55--59.

[36]

Skinner, G. K. 1988. X-Ray Imaging with Coded Masks. Scientific American 259 (Aug.), 84.

[37]

Sun, W., and Barbastathis, G. 2005. Rainbow volume holographic imaging. Opt. Lett. 30, 976--978.

[38]

Vaish, V., Wilburn, B., Joshi, N., and Levoy, M. 2004. Using plane + parallax for calibrating dense camera arrays. In Proc. Conf. Computer Vision and Pattern Recognition, 2--9.

[39]

van der Gracht, J., Dowski, E., Taylor, M., and Deaver, D. 1996. Broadband behavior of an optical-digital focus-invariant system. Optics Letters 21, 13 (July), 919--921.

[40]

Veeraraghavan, A., Raskar, R., Agrawal, A., Mohan, A., and Tumblin, J. 2007. Non-refractive modulators for coding and capturing scene appearance. Tech. Rep. UMIACS-TR-2007-21, Univ. of Maryland.

[41]

Wang, S., and Heidrich, W. 2004. The design of an inexpensive very high resolution scan camera system. Eurographics 23, 441--450.

[42]

Wilburn, B., Joshi, N., Vaish, V., Talvala, E.-V., Antunez, E., Barth, A., Adams, A., Horowitz, M., and Levoy, M. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3, 765--776.

Digital Library

[43]

Yang, J. C. 2000. A Light Field Camera For Image Based Rendering. Master's thesis, Massachussettes Institute of Technology.

[44]

Zomet, A., and Nayar, S. 2006. Lensless imaging with a controllable aperture. In Proc. Conf. Computer Vision and Pattern Recognition, 339--346.

Digital Library

Cited By

Lee JHo HYoo SKwon YSohn H(2022)Distant object localization with a single image obtained from a smartphone in an urban environmentInternational Journal of Applied Earth Observation and Geoinformation10.1016/j.jag.2022.102820111(102820)Online publication date: Jul-2022
https://doi.org/10.1016/j.jag.2022.102820
Rego JKulkarni KJayasuriya S(2021)Robust Lensless Image Reconstruction via PSF Estimation2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00045(403-412)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00045
Yang ASankaranarayanan A(2021)Designing Display Pixel Layouts for Under-Panel CamerasIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.307597843:7(2245-2256)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TPAMI.2021.3075978
Show More Cited By

Index Terms

Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision tasks
        Scene understanding
      2. Image and video acquisition
2. Theory of computation
  1. Randomness, geometry and discrete structures
    1. Computational geometry

Recommendations

Dappled photography: mask enhanced cameras for heterodyned light fields and coded aperture refocusing

We describe a theoretical framework for reversibly modulating 4D light fields using an attenuating mask in the optical path of a lens based camera. Based on this framework, we present a novel design to reconstruct the 4D light field from a 2D camera ...
Dual photography
SIGGRAPH '05: ACM SIGGRAPH 2005 Papers

We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate ...
Dual photography

We present a novel photographic technique called dual photography, which exploits Helmholtz reciprocity to interchange the lights and cameras in a scene. With a video projector providing structured illumination, reciprocity permits us to generate ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SIGGRAPH '07: ACM SIGGRAPH 2007 papers

August 2007

1019 pages

ISBN:9781450378369

DOI:10.1145/1275808

Conference Chair:
Marc Levoy

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGGRAPH: ACM Special Interest Group on Computer Graphics and Interactive Techniques

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 29 July 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Article

Conference

SIGGRAPH07

Sponsor:

SIGGRAPH

SIGGRAPH07: Special Interest Group on Computer Graphics and Interactive Techniques Conference

August 5 - 9, 2007

California, San Diego

Acceptance Rates

SIGGRAPH '07 Paper Acceptance Rate 108 of 455 submissions, 24%;

Overall Acceptance Rate 1,822 of 8,601 submissions, 21%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

107
Total Citations
View Citations
2,237
Total Downloads

Downloads (Last 12 months)9
Downloads (Last 6 weeks)0

Reflects downloads up to 03 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Lee JHo HYoo SKwon YSohn H(2022)Distant object localization with a single image obtained from a smartphone in an urban environmentInternational Journal of Applied Earth Observation and Geoinformation10.1016/j.jag.2022.102820111(102820)Online publication date: Jul-2022
https://doi.org/10.1016/j.jag.2022.102820
Rego JKulkarni KJayasuriya S(2021)Robust Lensless Image Reconstruction via PSF Estimation2021 IEEE Winter Conference on Applications of Computer Vision (WACV)10.1109/WACV48630.2021.00045(403-412)Online publication date: Jan-2021
https://doi.org/10.1109/WACV48630.2021.00045
Yang ASankaranarayanan A(2021)Designing Display Pixel Layouts for Under-Panel CamerasIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.307597843:7(2245-2256)Online publication date: 1-Jul-2021
https://doi.org/10.1109/TPAMI.2021.3075978
Miandji ENguyen HHajisharif SUnger JGuillemot C(2021)Compressive HDR Light Field Imaging Using a Single Multi-ISO SensorIEEE Transactions on Computational Imaging10.1109/TCI.2021.31321917(1369-1384)Online publication date: 2021
https://doi.org/10.1109/TCI.2021.3132191
Anwar SHayder ZPorikli F(2021)Deblur and deep depth from single defocus imageMachine Vision and Applications10.1007/s00138-020-01162-632:1Online publication date: 7-Jan-2021
https://doi.org/10.1007/s00138-020-01162-6
Liang J(2020)Punching holes in light: recent progress in single-shot coded-aperture optical imagingReports on Progress in Physics10.1088/1361-6633/abaf4383:11(116101)Online publication date: 27-Oct-2020
https://doi.org/10.1088/1361-6633/abaf43
Nagahara H(2019)Computational PhotographyコンピュテーショナルフォトグラフィThe Journal of The Institute of Image Information and Television Engineers10.3169/itej.73.46073:3(460-468)Online publication date: 2019
https://doi.org/10.3169/itej.73.460
Ajjanagadde GThrampoulidis CYedidia AWornell G(2019)Near-optimal Coded Apertures for Imaging via Nazarov’s TheoremICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)10.1109/ICASSP.2019.8682254(7690-7694)Online publication date: May-2019
https://doi.org/10.1109/ICASSP.2019.8682254
Alexander EGuo QKoppal SGortler SZickler T(2018)Focal FlowInternational Journal of Computer Vision10.1007/s11263-017-1051-5126:10(1062-1083)Online publication date: 1-Oct-2018
https://dl.acm.org/doi/10.1007/s11263-017-1051-5
Wahab AAlam MGunturk B(2017)High dynamic range imaging using a plenoptic camera2017 25th Signal Processing and Communications Applications Conference (SIU)10.1109/SIU.2017.7960660(1-4)Online publication date: May-2017
https://doi.org/10.1109/SIU.2017.7960660
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten