research-article

Over- and Under-Exposure Reconstruction of a Single Plenoptic Capture

Authors:

Erik ReinhardAuthors Info & Claims

ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), Volume 14, Issue 2

Article No.: 52, Pages 1 - 21

https://doi.org/10.1145/3199514

Published: 22 May 2018 Publication History

Abstract

Light field images, for example, taken with plenoptic cameras, offer interesting post-processing opportunities, including depth-of-field management, depth estimation, viewpoint selection, and 3D image synthesis. Like most capture devices, however, plenoptic cameras have a limited dynamic range, so that over- and under-exposed areas in plenoptic images are commonplace. We therefore present a straightforward and robust plenoptic reconstruction technique based on the observation that vignetting causes peripheral views to receive less light than central views. Thus, corresponding pixels in different views can be used to reconstruct illumination, especially in areas where information missing in one view is present in another. Our algorithm accurately reconstructs under- and over-exposed regions (known as declipping), additionally affording an increase in peak luminance by up to two f-stops, and a comparable lowering of the noise floor. The key advantages of this approach are that no hardware modifications are necessary to improve the dynamic range, that no multiple exposure techniques are required, and therefore that no ghosting or other artifacts are introduced.

References

[1]

Mekides Assefa Abebe, Tania Pouli, Jonathan Kervec, and Chaker Larabi. 2015. Color clipping and over-exposure correction. In Eurographics Symposium on Rendering—Experimental Ideas 8 Implementations, Jaakko Lehtinen and Derek Nowrouzezahrai (eds.). The Eurographics Association.

[2]

Edward H. Adelson and John Y. A. Wang. 1992. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2 (1992), 99--106.

Digital Library

[3]

Kurt Barton Akeley, Brian Cabral, Colvin Pitts, Chia-Kai Liang, Bennett Wilburn, Timothy James Knight, and Yi-Ren Ng. 2014. Compensation for sensor saturation and microlens modulation during light-field image processing. (2014). Patent No. US 9386288 B2, filed December 15, 2014, issued July 5, 2016.

[4]

Tom E. Bishop and Paolo Favaro. 2012. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34, 5 (2012), 972--986.

Digital Library

[5]

Tom E. Bishop, Sara Zanetti, and Paolo Favaro. 2009. Light field superresolution. In Proceedings of the IEEE International Conference on Computational Photography. 1--9.

[6]

D. Cho, M. Lee, S. Kim, and Y.-W. Tai. 2013. Modeling the calibration pipeline of the Lytro camera for high quality light-field image reconstruction. In IEEE International Conference on Computer Vision. 3280--3287.

Digital Library

[7]

F. R. Chung. 1997. Spectral Graph Theory. American Mathematical Society.

Digital Library

[8]

Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. 2004. Region filling and object removal by exemplar-based image inpainting. IEEE Trans. Image Process. 13, 9 (2004), 1200--1212.

Digital Library

[9]

Donald Dansereau, Oscar Pizarro, and Stefan Williams. 2013. Decoding, calibration and rectification for lenselet-based plenoptic cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 1027--1034.

Digital Library

[10]

Paul Debevec and Jitendra Malik. 1997. Recovering high dynamic range radiance maps from photographs. In SIGGRAPH’97: Proceedings of the 24th Annual Conference on Computer Graphics and Interactive Techniques. 369--378.

Digital Library

[11]

Alexei A. Efros and Thomas K. Leung. 1999. Texture synthesis by non-parametric sampling. In Proceedings of the Seventh IEEE International Conference on Computer Vision, 1999, Vol. 2. 1033--1038.

Digital Library

[12]

Juliet Fiss, Brian Curless, and Richard Szeliski. 2014. Refocussing plenoptic images using depth-adaptive splatting. In Proceedings of the IEEE International Conference on Computational Photography. 1--9.

[13]

Orazio Gallo, N. Gelfandz, Wei-Chao Chen, Marius Tico, and Kari Pulli. 2009. Artifact-free high dynamic range imaging. In Proceedings of the 2009 IEEE International Conference on Computational Photography (ICCP’09). IEEE, 1--7.

[14]

Elena Garces, Jose I. Echevarria, Wen Zhang, Hongzhi Wu, Kun Zhou, and Diego Gutierrez. 2016. Intrinsic light fields. https://arxiv.org/abs/1608.04342 (2016).

[15]

Todor Georgiev and Andrew Lumsdaine. 2010. Rich image capture with plenoptic cameras. In Proceedings of the IEEE International Conference on Computational Photography. 1--8.

[16]

Todor Georgiev, Andrew Lumsdaine, and Sergio Goma. 2009. High dynamic range image capture with plenoptic 2.0 camera. In Proceedings of Signal Recovery and Synthesis. Optical Society of America, SWA7P.

[17]

Todor Georgiev, Ke Colin Zheng, Brian Curless, David Salesin, Shree Nayar, and Chintan Intwala. 2006. Spatio-angular resolution tradeoff in integral photography. In Proceedings of the Eurographics Symposium on Rendering. 263--272.

Digital Library

[18]

Steven J. Gortler, Radek Grzeszczuk, Richard Szeliski, and Michael F. Cohen. 1996. The lumigraph. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, 43--54.

Digital Library

[19]

Miguel Granados, Hans-Peter Seidel, and Hendrik P. A. Lensch. 2008. Background estimation from non-time sequence images. In Proceedings of Graphics Interface 2008. 33--40.

Digital Library

[20]

Christopher Hahne, Amar Aggoun, Vladan Velisavljevic, Susanne Fiebig, and Matthias Pesch. 2016. Refocusing distance of a standard plenoptic camera. Optics Express 24, 19 (2016), 21521--21540.

[21]

Roarke Horstmeyer, Gary Euliss, Ravindra Athale, and Marc Levoy. 2009. Flexible multimodal camera using a light field architecture. In Proceedings of the IEEE Conference on Computational Photography. 1--8.

[22]

W. Hu, G. Cheung, and M. Kazui. 2016. Graph-based dequantization of block-compressed piecewise smooth images. IEEE Signal Process. Lett. 23, 2 (2016), 242--246.

[23]

W. Hu, X. Li, G. Cheung, and O. Au. 2013. Depth map denoising using graph-based transform and group sparsity. In Proceedings of the IEEE International Workshop on Multimedia Signal Processing.

[24]

Wei Hu, Mozhdeh Seifi, and Erik Reinhard. 2017. Optical center estimation for lenslet-based plenoptic cameras. In 19th IEEE International Workshop on Multimedia Signal Processing (MMSP’17).

[25]

Ivo Ihrke, Timo Stich, Heiko Gottschlich, Marcus Magnor, and Hans-Peter Seidel. 2008. Fast incident light field acquisition and rendering. J. WSCG 16, 1 (2008), 25--32.

[26]

Aaron Isaksen, Leonard McMillan, and Steven J Gortler. 2000. Dynamically reparameterized light fields. In Proceedings of the 27th Annual Conference on Computer Graphics and Interactive Techniques. 297--306.

Digital Library

[27]

Frederic E. Ives. 1903. Parallax Stereogram and Process of Making Same. (Filed Sept. 25, 1902, issued Apr. 14, 1903). U.S. Patent US725567 A.

[28]

E. A. Khan, A. O. Akyuz, and E. Reinhard. 2006. Ghost removal in high dynamic range images. In IEEE International Conference on Image Processing. 2005--2008.

[29]

Seon Joo Kim and Marc Pollefeys. 2008. Robust radiometric calibration and vignetting correction. IEEE Trans. Pattern Anal. Mach. Intell. 30, 4 (2008), 562--576.

Digital Library

[30]

Marc Levoy and Pat Hanrahan. 1996. Light field rendering. In Proceedings of the 23rd Annual Conference on Computer Graphics and Interactive Techniques. ACM, 31--42.

Digital Library

[31]

Marc Levoy, Ren Ng, Andrew Adams, Matthew Footer, and Mark Horowitz. 2006. Light field microscopy. ACM Trans. Graph. 25, 3 (2006), 924--934.

Digital Library

[32]

Chia-Kai Liang and Ravi Ramamoorthi. 2015. A light transport framework for lenslet light field cameras. ACM Trans. Graph. 34, 2 (March 2015), 16:1--16:19.

Digital Library

[33]

Andrew Lumsdaine and Todor Georgiev. 2008. Full Resolution Lightfield Rendering.Technical report, Indiana University and Adobe Systems.

[34]

Andrew Lumsdaine and Todor Georgiev. 2009. The focused plenoptic camera. In IEEE International Conference on Computational Photography (ICCP’09). 1--8.

[35]

U. Luxburg. 2007. A tutorial on spectral clustering. In Statistics and Computing, Vol. 17, No. 4. 395--416.

Digital Library

[36]

Henrique S. Malvar, Li-Wei He, and Ross Cutler. 2004. High-quality linear interpolation for demosaicking of bayer-patterned color images. In Proceedings of the IEEE International Conference on Acoustics, Speech, and Signal Processing, Vol. 3. iii--485.

[37]

Alkhazur Manakov, John Restrepo, Oliver Klehm, Ramon Hegedus, Elmar Eisemann, Hans-Peter Seidel, and Ivo Ihrke. 2013. A reconfigurable camera add-on for high dynamic range, multispectral, polarization, and light-field imaging. ACM Trans. Graph. 32, 4 (2013), 47--1.

Digital Library

[38]

S. Mann and R. W. Picard. 1995. On being ‘undigital’ with digital cameras: Extending dynamic range by combining differently exposed pictures. In Proceedings of IS&T 48th Annual Conference. 422--428.

[39]

Syed Z. Masood, Jiejie Zhu, and Marshall F. Tappen. 2009. Automatic correction of saturated regions in photographs using cross-channel correlation. Comput. Graph. Forum 28, 7 (2009), 1861--1869.

[40]

Vincent Masselus, Pieter Peers, Philip Dutré, and Yves D. Willems. 2003. Relighting with 4D incident light fields. ACM Trans. Graph. 22, 3 (2003), 613--620.

Digital Library

[41]

Peyman Milanfar. 2013. A tour of modern image filtering: New insights and methods, both practical and theoretical. IEEE Signal Process. Mag. 30, 1 (2013), 106--128.

[42]

T. Mitsunaga and S. K. Nayar. 1999. Radiometric self calibration. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Vol. 1. 374--380.

[43]

Shree K. Nayar and Srinivasa G. Narasimhan. 2002. Assorted pixels: Multisampled imaging with structural models. In Proceedings of the European Conference on Computer Vision. 636--652.

Digital Library

[44]

Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, and Pat Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. Comput. Sci. Tech. Rep. 2, 11 (2005), 1--11.

[45]

Christian Perwass and Lennart Wietzke. 2012. Single lens 3D-camera with extended depth-of-field. In IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, 829108--829108.

[46]

Colvin Pitts, Timothy James Knight, Chia-Kai Liang, and Yi-Ren Ng. 2015. Generating dolly zoom effect using light field image data. (2015). Patent No. US 8971625 B2, filed June 23, 2014, issued March 3, 2015.

[47]

Ramesh Raskar, Amit Agrawal, Cyrus A. Wilson, and Ashok Veeraraghavan. 2008. Glare aware photography: 4D Ray sampling for reducing glare effects of camera lenses. ACM Trans. Graph. 27, 3 (2008), 56:1--56:10.

Digital Library

[48]

E. Reinhard and K. Devlin. 2005. Dynamic range reduction inspired by photoreceptor physiology. IEEE Trans. Vis. Comput. Graph. 11, 1 (2005), 13--24.

Digital Library

[49]

E. Reinhard, W. Heidrich, P. Debevec, S. Pattanaik, G. Ward, and K. Myszkowski. 2010. High Dynamic Range Imaging: Acquisition, Display and Image-Based Lighting (2nd ed.). Morgan Kaufmann.

[50]

Erik Reinhard, Erum Arif Khan, Ahmet Oğuz Akyüz, and Garrett M. Johnson. 2008. Color Imaging: Fundamentals and Applications. A K Peters, Wellesley.

Digital Library

[51]

Erik Reinhard, Michael Stark, Peter Shirley, and James Ferwerda. 2002. Photographic tone reproduction for digital images. ACM Trans. Graph. 21, 3 (2002), 267--276.

Digital Library

[52]

M. A. Robertson, S. Borman, and R. L. Stevenson. 2003. Estimation-theoretic approach to dynamic range enhancement using multiple exposures. Journal of Electronic Imaging 12 (2003), 219--228. Issue 2.

[53]

Neus Sabater, Mozhdeh Seifi, Valter Drazic, Gustavo Sandri, and Patrick Pérez. 2014. Accurate disparity estimation for plenoptic images. In Computer Vision — ECCV 2014 Workshops. Springer, 548--560.

[54]

Yoav Y. Schechner and Shree K. Nayar. 2003. Generalized mosaicing: High dynamic range in a wide field of view. International Journal of Computer Vision 53, 3 (2003), 245--267.

Digital Library

[55]

David C. Schedl, Clemens Birklbauer, and Oliver Bimber. 2014. Coded exposure HDR light-field video recording. Computer Graphics Forum 33, 2 (2014), 33--42.

Digital Library

[56]

Pradeep Sen, Nima Khademi Kalantari, Maziar Yaesoubi, Soheil Darabi, Dan B. Goldman, and Eli Shechtman. 2012. Robust patch-based HDR reconstruction of dynamic scenes. ACM Trans. Graph. 31, 6 (2012), 203:1--203:11.

Digital Library

[57]

D. I. Shuman, S. K. Narang, P. Frossard, A. Ortega, and P. Vandergheynst. 2013. The emerging field of signal processing on graphs. IEEE Signal Processing Magazine 30, 3 (2013), 83--98.

[58]

Thiow Keng Tan, Choong Seng Boon, and Yuya Suzuki. 2006. Intra prediction by template matching. In Proceedings of the IEEE International Conference on Image Processing. 1693--1696.

[59]

Michael Tao, Sunil Hadap, Jitendra Malik, and Ravi Ramamoorthi. 2013. Depth from combining defocus and correspondence using light-field cameras. In Proceedings of the IEEE International Conference on Computer Vision. 673--680.

Digital Library

[60]

J. Unger, S. Gustavson, P. Larsson, and A. Ynnerman. 2008. Free form incident light fields. Computer Graphics Forum 27, 4 (2008), 1293--1301.

Digital Library

[61]

J. Unger, A. Wenger, T. Hawtins, A. Gardner, and P. Debevec. 2003. Capturing and rendering with incident light fields. In Proceedings of the Eurographics Symposium on Rendering. 141--149.

Digital Library

[62]

Ashok Veeraraghavan, Ramesh Raskar, Amit Agrawal, Ankit Mohan, and Jack Tumblin. 2007. Dappled photography: Mask enhanced cameras for hyterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3 (2007), 69.

Digital Library

[63]

Lvdi Wang, Li-Yi Wei, Kun Zhou, Baining Guo, and Heung-Yeung Shum. 2007. High dynamic range image hallucination. In Proceedings of the 18th Eurographics conference on Rendering Techniques. Eurographics Association, 321--326.

Digital Library

[64]

Ting-Chun Wang, Jun-Yan Zhu, Ebi Hiroaki, Manmohan Chandraker, Alexei A Efros, and Ravi Ramamoorthi. 2016. A 4D light-field dataset and CNN architecture for material recognition. In Proceedings of the European Conference on Computer Vision. 121--138.

[65]

Zhou Wang, Alan C. Bovik, Hamid R. Sheikh, and Eero P. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4 (2004), 600--612.

Digital Library

[66]

Sven Wanner and Bastian Goldluecke. 2014. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36, 3 (2014), 606--619.

Digital Library

[67]

Greg Ward. 2003. Fast, robust image registration for compositing high dynamic range photographs from hand-held exposures. J. Graph. Tools 8, 2 (2003), 17--30.

[68]

Li-Yi Wei and Marc Levoy. 2001. Texture synthesis over arbitrary manifold surfaces. In SIGGRAPH’01: Proceedings of the 28th Annual Conference on Computer Graphics and Interactive Techniques. 355--360.

Digital Library

[69]

Z. Yu, X. Guo, H. Ling, A. Lumsdaine, and J. Yu. 2013. Line assisted light field triangulation and stereo matching. In Proceedings of the IEEE International Conference on Computer Vision. 2792--2799.

Digital Library

[70]

Zhan Yu, Jingji Yu, Andrew Lumsdaine, and Todor Georgiev. 2012. An analysis of color demosaicing in plenoptic cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 901--908.

Digital Library

[71]

Yuanjie Zheng, Stephen Lin, Chandra Kambhamettu, Jingyi Yu, and Sing Bing Kang. 2009. Single-image vignetting correction. IEEE Trans. Pattern Anal. Mach. Intell. 31, 12 (2009), 2243--2256.

Digital Library

[72]

Shan Zhu and Kai-Kuang Ma. 2000. A new diamond search algorithm for fast block-matching motion estimation. IEEE Trans. Image Process. 9, 2 (2000), 287--290.

Digital Library

Cited By

Li WCao TLiu CTian XLi YWang XDong X(2023)Dual-Lens HDR using Guided 3D Exposure CNN and Guided Denoising TransformerACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357916719:5(1-20)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3579167
Tang LHuang HZhang YQi GYu Z(2023)Structure-embedded ghosting artifact suppression network for high dynamic range image reconstructionKnowledge-Based Systems10.1016/j.knosys.2023.110278263(110278)Online publication date: Mar-2023
https://doi.org/10.1016/j.knosys.2023.110278
Fu ZHu WGuo Z(2020)3d Dynamic Point Cloud Inpainting Via Temporal Consistency On Graphs2020 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME46284.2020.9102861(1-6)Online publication date: Jul-2020
https://doi.org/10.1109/ICME46284.2020.9102861
Show More Cited By

Index Terms

Over- and Under-Exposure Reconstruction of a Single Plenoptic Capture
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision representations
        Image representations
      2. Image and video acquisition
        Computational photography
  2. Computer graphics
    1. Image manipulation
      1. Computational photography
      2. Image processing

Recommendations

RII: Renovating the irregular illumination of digital image archives

Digitization is critical for preserving valuable culture archives such as texts and images. Considering the physical characters of digital cameras or scanners and the artificial negligence, some distortions such as under-exposure or over-exposure often ...
Layered Light Field Reconstruction for Defocus Blur

We present a novel algorithm for reconstructing high-quality defocus blur from a sparsely sampled light field. Our algorithm builds upon recent developments in the area of sheared reconstruction filters and significantly improves reconstruction quality ...
Plenoptic sampling
SIGGRAPH '00: Proceedings of the 27th annual conference on Computer graphics and interactive techniques

This paper studies the problem of plenoptic sampling in image-based rendering (IBR). From a spectral analysis of light field signals and using the sampling theorem, we mathematically derive the analytical functions to determine the minimum sampling rate ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Multimedia Computing, Communications, and Applications

ACM Transactions on Multimedia Computing, Communications, and Applications Volume 14, Issue 2

May 2018

208 pages

ISSN:1551-6857

EISSN:1551-6865

DOI:10.1145/3210458

Editor:
Alberto Del Bimbo
University of Firenze, Italy

Issue’s Table of Contents

Copyright © 2018 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 May 2018

Accepted: 01 March 2018

Revised: 01 February 2018

Received: 01 August 2017

Published in TOMM Volume 14, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

National Natural Science Foundation of China
MSRA Collaborative Research
Alibaba Innovative Research

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

5
Total Citations
View Citations
141
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li WCao TLiu CTian XLi YWang XDong X(2023)Dual-Lens HDR using Guided 3D Exposure CNN and Guided Denoising TransformerACM Transactions on Multimedia Computing, Communications, and Applications10.1145/357916719:5(1-20)Online publication date: 16-Mar-2023
https://dl.acm.org/doi/10.1145/3579167
Tang LHuang HZhang YQi GYu Z(2023)Structure-embedded ghosting artifact suppression network for high dynamic range image reconstructionKnowledge-Based Systems10.1016/j.knosys.2023.110278263(110278)Online publication date: Mar-2023
https://doi.org/10.1016/j.knosys.2023.110278
Fu ZHu WGuo Z(2020)3d Dynamic Point Cloud Inpainting Via Temporal Consistency On Graphs2020 IEEE International Conference on Multimedia and Expo (ICME)10.1109/ICME46284.2020.9102861(1-6)Online publication date: Jul-2020
https://doi.org/10.1109/ICME46284.2020.9102861
Hu WFu ZGuo Z(2019)Local Frequency Interpretation and Non-Local Self-Similarity on Graph for Point Cloud InpaintingIEEE Transactions on Image Processing10.1109/TIP.2019.290655428:8(4087-4100)Online publication date: Aug-2019
https://doi.org/10.1109/TIP.2019.2906554
Pan CZhou LLuo YWang XWang Z(2019)Lightweight and Accurate Memory Allocation in Key-Value CacheInternational Journal of Parallel Programming10.1007/s10766-018-0616-447:3(451-466)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s10766-018-0616-4

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents