research-article

Linear Volumetric Focus for Light Field Cameras

Authors:

Donald G. Dansereau,

Stefan B. WilliamsAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 2

Article No.: 15, Pages 1 - 20

https://doi.org/10.1145/2665074

Published: 02 March 2015 Publication History

Abstract

We demonstrate that the redundant information in light field imagery allows volumetric focus, an improvement of signal quality that maintains focus over a controllable range of depths. To do this, we derive the frequency-domain region of support of the light field, finding it to be the 4D hyperfan at the intersection of a dual fan and a hypercone, and design a filter with correspondingly shaped passband. Drawing examples from the Stanford Light Field Archive and images captured using a commercially available lenslet-based plenoptic camera, we demonstrate that the hyperfan outperforms competing methods including planar focus, fan-shaped antialiasing, and nonlinear image and video denoising techniques. We show the hyperfan preserves depth of field, making it a single-step all-in-focus denoising filter suitable for general-purpose light field rendering. We include results for different noise types and levels, through murky water and particulate matter, in real-world scenarios, and evaluated using a variety of metrics. We show that the hyperfan's performance scales with aperture count, and demonstrate the inclusion of aliased components for high-quality rendering.

Supplementary Material

JPG File (a15.jpg)

Download
19.05 KB

MP4 File (a15.mp4)

Download
70.57 MB

References

[1]

E. H. Adelson and J. Y. A. WANG. 2002. Single lens stereo with a plenoptic camera. IEEE Trans. Pattern Anal. Mach. Intell. 14, 2, 99--106.

Digital Library

[2]

A. Agrawal, Y. Xu, and R. Raskar. 2009. Invertible motion blur in video. ACM Trans. Graph. 28, 3.

Digital Library

[3]

M. Aharon, M. Elad, and A. Bruckstein. 2006. K-SVD: An algorithm for designing overcomplete dictionaries for sparse representation. IEEE Trans. Signal Process. 54, 11, 4311--4322.

Digital Library

[4]

R. Ansari. 1987. Efficient IIR and FIR fan filters. IEEE Trans. Circ. Syst. 34, 8, 941--945.

[5]

J. Berent and P. L. Dragotti. 2007. Plenoptic manifolds. Signal Process. Mag. 24, 6, 34--44.

[6]

T. E. Bishop and P. Favaro. 2012. The light field camera: Extended depth of field, aliasing, and superresolution. IEEE Trans. Pattern Anal. Mach. Intell. 34, 5, 972--986.

Digital Library

[7]

K. Bitsakos and C. Fermuller. 2006. Depth estimation using the compound eye of dipteran flies. Biol. Cybernet. 95, 5, 487--501.

Digital Library

[8]

R. Bolles, H. Baker, and D. Marimont. 1987. Epipolar-plane image analysis: An approach to determining structure from motion. Intl. J. Comput. Vis. 1, 1, 7--55.

[9]

A. Buades, B. Coll, and J.-M. Morel. 2005. A review of image denoising algorithms, with a new one. SIAM J. Multiscale Model. Simul. 4, 2, 490--530.

[10]

A. Cetin, O. Gerek, and Y. Yardimci. 1997. Equiripple FIR filter design by the FFT algorithm. IEEE Signal Process. Mag. 14, 2, 60--64.

[11]

J. Chai, X. Tong, S. Chan, and H. Shum. 2000. Plenoptic sampling. In Proceedings of the 27^th Annual Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). 307--318.

Digital Library

[12]

S.-C. Chan and H.-Y. Shum. 2000. A spectral analysis for light field rendering. In Proceedings of the International Conference on Image Processing (ICIP'00). Vol. 2, 25--28.

[13]

O. Cossairt, M. Gupta, and S. Nayar. 2012. When does computational imaging improve performance&quest; IEEE Trans. Image Process. 22, 2, 447--458.

Digital Library

[14]

K. Dabov, A. Foi, and K. Egiazarian. 2007. Video denoising by sparse 3D transform-domain collaborative filtering. In Proceedings of the 15^th European Signal Processing Conference (EURIPCO'07). 7.

[15]

K. J. Dana, B. Van Ginneken, S. K. Nayar, and J. J. Koenderink. 1999. Reflectance and texture of real-world surfaces. ACM Trans. Graph. 18, 1, 1--34.

Digital Library

[16]

D. G. Dansereau. 2003. 4D light field processing and its application to computer vision. M.S. thesis, Department of Electrical and Computer Engineering, University of Calgary. http://www-personal.acfr.usyd.edu.au/ddan1654/dansereau2003MscThesis.pdf.

[17]

D. G. Dansereau. 2014. Plenoptic signal processing for robust vision in field robotics. Ph.D. thesis, Australian Centre for Field Robotics, School of Aerospace, Mechanical and Mechatronic Engineering, The University of Sydney.

[18]

D. G. Dansereau, D. L. Bongiorno, O. Pizarro, and S. B. Williams. 2013a. Light field image denoising using a linear 4D frequency-hyperfan all-in-focus filter. In Proceedings of the SPIE Conference on Computational Imaging (SPIE'13). Vol. 8657.

[19]

D. G. Dansereau, O. Pizarro, and S. B. Williams. 2013b. Decoding, calibration and rectification for lenselet-based plenoptic cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13). 1027--1034.

Digital Library

[20]

D. G. Dansereau and L. T. Bruton. 2003. A 4D frequency-planar IIR filter and its application to light field processing. In Proceedings of the International Symposium on Circuits and Systems (ISCAS'03). Vol. 4. 476--479.

[21]

D. G. Dansereau and L. T. Bruton. 2004. Gradient-based depth estimation from 4D light fields. In Proceedings of the International Symposium on Circuits and Systems (ISCAS'04). Vol. 3, 549--552.

[22]

D. G. Dansereau and L. T. Bruton. 2007. A 4-D dual-fan filter bank for depth filtering in light fields. IEEE Trans. Signal Process. 55, 2, 542--549.

[23]

D. G. Dansereau, I. Mahon, O. Pizarro, and S. B. Williams. 2011. Plenoptic flow: Closed-form visual odometry for light field cameras. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'11). 4455--4462.

[24]

D. G. Dansereau and S. B. Williams. 2011. Seabed modeling and distractor extraction for mobile AUVS using light field filtering. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA'11). 1634--1639.

[25]

F. Durand, N. Holzschuch, C. Soler, E. Chan, and F. Sillion. 2005. A frequency analysis of light transport. In Proceedings of the 32^nd International Conference on Computer Graphics and Interactive Techniques: Papers (SIGGRAPH'05). 1115--1126.

Digital Library

[26]

M. Elad and M. Aharon. 2006. Image denoising via sparse and redundant representations over learned dictionaries. IEEE Trans. Image Process. 15, 12, 3736--3745.

Digital Library

[27]

W. Freeman, A. Levin, S. Hasinoff, P. Green, and F. Durand. 2009. 4D frequency analysis of computational cameras for depth of field extension. Tech. rep. MIT-CSAIL-TR-2009-019. http://dspace.mit.edu/handle/1721.1/45513.

[28]

W. S. Geisler. 2008. Visual perception and the statistical properties of natural scenes. Ann. Rev. Psychol. 59, 167--192.

[29]

B. Goldluecke and S. Wanner. 2013. The variational structure of disparity and regularization of 4D light fields. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13). 1003--1010.

Digital Library

[30]

X. Gu, S. J. Gortler, and M. F. Cohen. 1997. Polyhedral geometry and the two-plane parameterization. In Proceedings of the Eurographics Workshop on Rendering Techniques. Springer, 1--12.

Digital Library

[31]

O. G. Guleryuz. 2007. Weighted averaging for denoising with overcomplete dictionaries. IEEE Trans. Image Process. 16, 12, 3020--3034.

Digital Library

[32]

A. Isaksen, L. McMillan, and S. Gortler. 2000. Dynamically reparameterized light fields. In Proceedings of the International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'00). 297--306.

Digital Library

[33]

Y. Ji, J. Ye, and J. Yu. 2013. Reconstructing gas flows using light-path approximation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13). 2507--2514.

Digital Library

[34]

J.-H. Lambert. 1760. Photometria, sive de Mensura et gradibus luminis, colorum et umbrae. Eberhard Klett.

[35]

A. Levin and F. Durand. 2010. Linear view synthesis using a dimensionality gap light field prior. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10). 1831--1838.

[36]

A. Levin, S. Hasinoff, P. Green, F. Durand, and W. Freeman. 2009. 4D frequency analysis of computational cameras for depth of field extension. ACM Trans. Graph. 28, 3.

Digital Library

[37]

M. Levoy, B. Chen, V. Vaish, M. Horowitz, I. McDowall, and M. Bolas. 2004. Synthetic aperture confocal imaging. ACM Trans. Graph. 23, 3, 825--834.

Digital Library

[38]

M. Levoy and P. Hanrahan. 1996. Light field rendering. In Proceedings of the International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'96). 31--42.

Digital Library

[39]

A. Lumsdaine and T. Georgiev. 2008. Full resolution lightfield rendering. Tech. rep., Adobe Systems. http://tgeorgiev.net/FullResolution.pdf.

[40]

A. Lumsdaine and T. Georgiev. 2009. The focused plenoptic camera. In Proceedings of the IEEE International Conference on Computational Photography (ICCP'09). 1--8.

[41]

K. Maeno, H. Nagahara, A. Shimada, and R.-I. Taniguchi. 2013. Light field distortion feature for transparent object recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'13). 2786--2793.

Digital Library

[42]

K. Mitra, O. Cossairt, and A. Veeraraghavan. 2013. A frame-work for the analysis of computational imaging systems with practical applications. http://arxiv.org/pdf/1308.1981.pdf.

[43]

H. Nagahara, S. Kuthirummal, C. Zhou, and S. K. Nayar. 2008. Flexible depth of field photography. In Proceedings of the European Conference on Computer Vision (ECCV'08). Springer, 60--73.

Digital Library

[44]

J. Neumann, C. Fermuller, Y. Aloimonos, and V. Brajovic. 2005. Compound eye sensor for 3D ego motion estimation. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS'05). Vol. 4. 3712--3717.

[45]

R. Ng. 2005. Fourier slice photography. ACM Trans. Graph. 24, 3, 735--744.

Digital Library

[46]

R. Ng, M. Levoy, M. Bredif, G. Duval, M. Horowitz, and P. Hanrahan. 2005. Light field photography with a hand-held plenoptic camera. Tech. rep. CSTR 2, Department of Computer Science, Stanford University. https://graphics.stanford.edu/papers/lfcamera/lfcamera-150dpi.pdf.

[47]

M. O'TOOLE, R. Raskar, and K. N. Kutulakos. 2012. Primal-dual coding to probe light transport. In Proceedings of the International Conference on Computer Graphics and Interactive Techniques (SIGGRAPH'12). Vol. 31. 39.

Digital Library

[48]

J. G. Proakis and M. Salehi. 2007. Digital Communications 5^th Ed. McGraw-Hill.

[49]

L. R. Rabiner and B. Gold. 1975. Theory and Application of Digital Signal Processing, Volume 1. Prentice-Hall.

[50]

R. Raskar, A. Agrawal, and J. Tumblin. 2006. Coded exposure photography: Motion deblurring using fluttered shutter. ACM Trans. Graph. 25, 3, 795--804.

Digital Library

[51]

R. Raskar, A. Agrawal, C. A. Wilson, and A. Veeraraghavan. 2008. Glare aware photography: 4D ray sampling for reducing glare effects of camera lenses. ACM Trans. Graph. 27, 3.

Digital Library

[52]

D. L. Ruderman. 1997. Origins of scaling in natural images. Vis. Res. 37, 23, 3385--3398.

[53]

A. Shnayderman, A. Gusev, and A. Eskicioglu. 2006. An SVD-based grayscale image quality measure for local and global assessment. IEEE Trans. Image Process. 15, 2, 422--429.

Digital Library

[54]

J. Stewart, J. Yu, S. J. Gortler, and L. McMillan. 2003. A new reconstruction filter for undersampled light fields. In Proceedings of the 14^th Eurographics Workshop on Rendering (EGRW'03). Eurographics Association, 150--156.

Digital Library

[55]

V. Vaish, M. Levoy, R. Szeliski, C. Zitnick, and S. Kang. 2006. Reconstructing occluded surfaces using synthetic apertures: Stereo, focus and robust measures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'06). Vol. 2. 2331--2338.

Digital Library

[56]

A. Veeraraghavan, R. Raskar, A. Agrawal, A. Mohan, and J. Tumblin. 2007. Dappled photography: Mask enhanced cameras for heterodyned light fields and coded aperture refocusing. ACM Trans. Graph. 26, 3.

Digital Library

[57]

Z. Wang, A. Bovik, H. Sheikh, and E. Simoncelli. 2004. Image quality assessment: From error visibility to structural similarity. IEEE Trans. Image Process. 13, 4, 600--612.

Digital Library

[58]

S. Wanner and B. Goldluecke. 2013. Variational light field analysis for disparity estimation and super-resolution. IEEE Trans. Pattern Anal. Mach. Intell. 36, 3, 606--619.

Digital Library

[59]

B. Wilburn, N. Joshi, V. Vaish, E. Talvala, E. Antunez, A. Barth, A. Adams, M. Horowitz, and M. Levoy. 2005. High performance imaging using large camera arrays. ACM Trans. Graph. 24, 3, 765--776.

Digital Library

[60]

H. Yang, M. Pollefeys, G. Welch, J. Frahm, and A. Ilie. 2007. Differential camera tracking through linearizing the local appearance manifold. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'07). 1--8.

[61]

Z. Yu, X. Guo, X. Chen, and Y. Yu. 2013. Catadioptric array photography for low light imaging. In Proceedings of the 2^nd IEEE International Workshop on Computational Cameras and Displays (CCD'13).

Cited By

ZHANG Runnan 张ZHOU Ning 周ZHOU Zihao 周DU Heheng 杜CHEN Qian 陈ZUO Chao 左(2024)光场表征及其分辨率提升技术：文献综述及最新进展(特邀)Infrared and Laser Engineering10.3788/IRLA2024034753:9(20240347)Online publication date: 2024
https://doi.org/10.3788/IRLA20240347
Mo CLiu XTong JXi JYu YCai Z(2024)Lensless light-field imaging using LMIOptics Express10.1364/OE.53902132:22(38112)Online publication date: 7-Oct-2024
https://doi.org/10.1364/OE.539021
Liu YQi ZCheng JChen X(2024)Rethinking the Effectiveness of Objective Evaluation Metrics in Multi-Focus Image Fusion: A Statistic-Based ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336790546:8(5806-5819)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3367905
Show More Cited By

Index Terms

Linear Volumetric Focus for Light Field Cameras
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Image and video acquisition
        Epipolar geometry
  2. Computer graphics
    1. Image manipulation

Recommendations

Hand-held 3D light field photography and applications

We propose a method to acquire 3D light fields using a hand-held camera, and describe several computational photography applications facilitated by our approach. As our input we take an image sequence from a camera translating along an approximately ...
Axial-cones: modeling spherical catadioptric cameras for wide-angle light field rendering

Catadioptric imaging systems are commonly used for wide-angle imaging, but lead to multi-perspective images which do not allow algorithms designed for perspective cameras to be used. Efficient use of such systems requires accurate geometric ray modeling ...
Axial-cones: modeling spherical catadioptric cameras for wide-angle light field rendering
SIGGRAPH ASIA '10: ACM SIGGRAPH Asia 2010 papers

Catadioptric imaging systems are commonly used for wide-angle imaging, but lead to multi-perspective images which do not allow algorithms designed for perspective cameras to be used. Efficient use of such systems requires accurate geometric ray modeling ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 2

February 2015

136 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2742222

Editor:
Holly Rushmeier
Yale University

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 02 March 2015

Accepted: 01 August 2014

Revised: 01 June 2014

Received: 01 March 2014

Published in TOG Volume 34, Issue 2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

University of Sydney
Australian Centre for Field Robotics
New South Wales State Government
Australian Government's International Postgraduate Research Scholarship and Endeavour Research Fellowship

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

119
Total Citations
View Citations
1,696
Total Downloads

Downloads (Last 12 months)33
Downloads (Last 6 weeks)5

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

ZHANG Runnan 张ZHOU Ning 周ZHOU Zihao 周DU Heheng 杜CHEN Qian 陈ZUO Chao 左(2024)光场表征及其分辨率提升技术：文献综述及最新进展(特邀)Infrared and Laser Engineering10.3788/IRLA2024034753:9(20240347)Online publication date: 2024
https://doi.org/10.3788/IRLA20240347
Mo CLiu XTong JXi JYu YCai Z(2024)Lensless light-field imaging using LMIOptics Express10.1364/OE.53902132:22(38112)Online publication date: 7-Oct-2024
https://doi.org/10.1364/OE.539021
Liu YQi ZCheng JChen X(2024)Rethinking the Effectiveness of Objective Evaluation Metrics in Multi-Focus Image Fusion: A Statistic-Based ApproachIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.336790546:8(5806-5819)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3367905
Wu WJin LLv ZLi GLi J(2024)Anti-Noise Light Field Depth Estimation Using Inline Occlusion HandlingIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2024.337820773(1-14)Online publication date: 2024
https://doi.org/10.1109/TIM.2024.3378207
Satheeskanth NWijenayake CEdussooriya C(2024)Ultra Low-Complexity 4-D IIR Hyperfan Filter for Light Field Denoising2024 Moratuwa Engineering Research Conference (MERCon)10.1109/MERCon63886.2024.10688959(736-741)Online publication date: 8-Aug-2024
https://doi.org/10.1109/MERCon63886.2024.10688959
Chekakta ZAouf N(2024)CaDNET: An End-to-End Plenoptic Camera-Based Deep Learning Pose Estimation Approach for Space Orbital RendezvousIEEE Sensors Journal10.1109/JSEN.2024.343574824:18(29441-29451)Online publication date: 15-Sep-2024
https://doi.org/10.1109/JSEN.2024.3435748
Kremer RHerfet T(2024)ST-SAIL: Spatio-temporal Semantic Analysis of Light Fields: Optimizing the Sampling Pattern of Light Field Arrays2024 IEEE International Conference on Consumer Electronics (ICCE)10.1109/ICCE59016.2024.10444447(1-6)Online publication date: 6-Jan-2024
https://doi.org/10.1109/ICCE59016.2024.10444447
Huang HCui CJia CZhang XMa S(2024)Lightweight macro-pixel quality enhancement network for light field images compressed by versatile video codingJournal of Visual Communication and Image Representation10.1016/j.jvcir.2024.104329(104329)Online publication date: Oct-2024
https://doi.org/10.1016/j.jvcir.2024.104329
Liu YXu JYee A(2024)The analysis of Iris image acquisition and real-time detection system using convolutional neural networkThe Journal of Supercomputing10.1007/s11227-023-05629-x80:4(4500-4532)Online publication date: 1-Mar-2024
https://dl.acm.org/doi/10.1007/s11227-023-05629-x
Wang Tiantian 王He Di 何Liu Chang 刘Qiu Jun 邱(2023)基于频谱集中程度的重参数化在光场去噪中的应用Acta Optica Sinica10.3788/AOS23065943:20(2026001)Online publication date: 2023
https://doi.org/10.3788/AOS230659
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents