Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
research-article

Learned large field-of-view imaging with thin-plate optics

Published: 08 November 2019 Publication History

Abstract

Typical camera optics consist of a system of individual elements that are designed to compensate for the aberrations of a single lens. Recent computational cameras shift some of this correction task from the optics to post-capture processing, reducing the imaging optics to only a few optical elements. However, these systems only achieve reasonable image quality by limiting the field of view (FOV) to a few degrees - effectively ignoring severe off-axis aberrations with blur sizes of multiple hundred pixels.
In this paper, we propose a lens design and learned reconstruction architecture that lift this limitation and provide an order of magnitude increase in field of view using only a single thin-plate lens element. Specifically, we design a lens to produce spatially shift-invariant point spread functions, over the full FOV, that are tailored to the proposed reconstruction architecture. We achieve this with a mixture PSF, consisting of a peak and and a low-pass component, which provides residual contrast instead of a small spot size as in traditional lens designs. To perform the reconstruction, we train a deep network on captured data from a display lab setup, eliminating the need for manual acquisition of training data in the field. We assess the proposed method in simulation and experimentally with a prototype camera system.
We compare our system against existing single-element designs, including an aspherical lens and a pinhole, and we compare against a complex multielement lens, validating high-quality large field-of-view (i.e. 53°) imaging performance using only a single thin-plate element.

Supplementary Material

ZIP File (a219-peng.zip)
Supplemental files.
MPG File (a219-peng.mpg)

References

[1]
Se Hyun Ahn and L Jay Guo. 2009. Large-area roll-to-roll and roll-to-plate nanoimprint lithography: a step toward high-throughput application of continuous nanoimprinting. ACS Nano 3, 8 (2009), 2304--2310.
[2]
Nick Antipa, Grace Kuo, Reinhard Heckel, Ben Mildenhall, Emrah Bostan, Ren Ng, and Laura Waller. 2018. DiffuserCam: lensless single-exposure 3D imaging. Optica 5, 1 (2018), 1--9.
[3]
Martin Arjovsky, Soumith Chintala, and Léon Bottou. 2017. Wasserstein gan. arXiv preprint arXiv:1701.07875 (2017).
[4]
Yoshua Bengio. 2012. Deep learning of representations for unsupervised and transfer learning. In Proceedings of ICML Workshop on Unsupervised and Transfer Learning. 17--36.
[5]
Glenn D Boreman. 2001. Modulation transfer function in optical and electro-optical systems. Vol. 21. SPIE press Bellingham, WA.
[6]
David J Brady, Michael E Gehm, Ronald A Stack, Daniel L Marks, David S Kittle, Dathon R Golish, EM Vera, and Steven D Feller. 2012. Multiscale gigapixel photography. Nature 486, 7403 (2012), 386.
[7]
Tim Brooks, Ben Mildenhall, Tianfan Xue, Jiawen Chen, Dillon Sharlet, and Jonathan T Barron. 2018. Unprocessing Images for Learned Raw Denoising. arXiv preprint arXiv:1811.11127 (2018).
[8]
Julie Chang, Vincent Sitzmann, Xiong Dun, Wolfgang Heidrich, and Gordon Wetzstein. 2018b. Hybrid optical-electronic convolutional neural networks with optimized diffractive optics for image classification. Scientific reports 8, 1 (2018), 12324.
[9]
Julie Chang and Gordon Wetzstein. 2019. Deep Optics for Monocular Depth Estimation and 3D Object Detection. arXiv preprint arXiv:1904.08601 (2019).
[10]
Li-Wen Chang, Yang Chen, Wenlei Bao, Amit Agarwal, Eldar Akchurin, Ke Deng, and Emad Barsoum. 2018a. Accelerating Recurrent Neural Networks through Compiler Techniques and Quantization. (2018).
[11]
Chen Chen, Qifeng Chen, Jia Xu, and Vladlen Koltun. 2018. Learning to See in the Dark. (2018).
[12]
Taeg Sang Cho, Charles L Zitnick, Neel Joshi, Sing Bing Kang, Richard Szeliski, and William T Freeman. 2012. Image restoration by matching gradient distributions. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI) 34, 4 (2012), 683--694.
[13]
Stephen Y Chou, Peter R Krauss, and Preston J Renstrom. 1996. Nanoimprint lithography. Journal of Vacuum Science & Technology B: Microelectronics and Nanometer Structures Processing, Measurement, and Phenomena 14, 6 (1996), 4129--4133.
[14]
Oliver Cossairt and Shree Nayar. 2010. Spectral focal sweep: Extended depth of field from chromatic aberrations. In Computational Photography (ICCP), IEEE International Conference on. IEEE, 1--8.
[15]
Edward R Dowski and W Thomas Cathey. 1995. Extended depth of field through wave-front coding. Applied optics 34, 11 (1995), 1859--1866.
[16]
Wang Duoshu, Chongtai Luo, Yuqing Xiong, Tao Chen, Hongkai Liu, and Jizhou Wang. 2011. Fabrication technology of the centrosymmetric continuous relief diffractive optical elements. Physics Procedia 18 (2011), 95--99.
[17]
EMVA Standard. 2005. 1288: Standard for characterization and presentation of specification data for image sensors and cameras. European Machine Vision Association (2005).
[18]
Magali Estribeau and Pierre Magnan. 2004. Fast MTF measurement of CMOS imagers using ISO 12333 slanted-edge methodology. In Detectors and Associated Signal Processing, Vol. 5251. International Society for Optics and Photonics, 243--253.
[19]
FZ Fang, XD Zhang, A Weckenmann, GX Zhang, and C Evans. 2013. Manufacturing and measurement of freeform optics. CIRP Annals 62, 2 (2013), 823--846.
[20]
Grant R Fowles. 2012. Introduction to modern optics. Courier Dover Publications.
[21]
Carl Friedrich Gauss. 1843. Dioptrische Untersuchungen von CF Gauss. in der Dieterichschen Buchhandlung.
[22]
Joseph M Geary. 2002. Introduction to lens design: with practical ZEMAX examples. Willmann-Bell Richmond.
[23]
Marc Geese, Ulrich Seger, and Alfredo Paolillo. 2018. Detection Probabilities: Performance Prediction for Sensors of Autonomous Vehicles. Electronic Imaging 2018, 17 (2018), 148-1--148-14.
[24]
Patrice Genevet, Federico Capasso, Francesco Aieta, Mohammadreza Khorasaninejad, and Robert Devlin. 2017. Recent advances in planar optics: from plasmonic to dielectric metasurfaces. Optica 4, 1 (2017), 139--152.
[25]
Erez Gilad and Jost Von Hardenberg. 2006. A fast algorithm for convolution integrals with space and time variant kernels. J. Comput. Phys. 216, 1 (2006), 326--336.
[26]
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in neural information processing systems. 2672--2680.
[27]
Joseph W Goodman. 2005. Introduction to Fourier optics. Roberts and Company Publishers.
[28]
Hrel Haim, Shay Elmalem, Raja Giryes, Alex M. Bronstein, and Emanuel Marom. 2018. Depth Estimation From a Single Image Using Deep Learned Phase Coded Mask. IEEE Trans. Computational Imaging 4, 3 (2018), 298--310.
[29]
Samuel W Hasinoff, Dillon Sharlet, Ryan Geiss, Andrew Adams, Jonathan T Barron, Florian Kainz, Jiawen Chen, and Marc Levoy. 2016. Burst photography for high dynamic range and low-light imaging on mobile cameras. ACM Transactions on Graphics (TOG) 35, 6 (2016), 192.
[30]
Eugene Hecht. 1998. Hecht optics. Addison Wesley 997 (1998), 213--214.
[31]
Felix Heide, Qiang Fu, Yifan Peng, and Wolfgang Heidrich. 2016. Encoded diffractive optics for full-spectrum computational imaging. Scientific Reports 6 (2016).
[32]
Felix Heide, Mushfiqur Rouf, Matthias B Hullin, Bjorn Labitzke, Wolfgang Heidrich, and Andreas Kolb. 2013. High-quality computational imaging through simple lenses. ACM Transactions on Graphics (TOG) 32, 5 (2013), 149.
[33]
Felix Heide, Markus Steinberger, Yun-Ta Tsai, Mushfiqur Rouf, Dawid Pająk, Dikpal Reddy, Orazio Gallo, Jing Liu, Wolfgang Heidrich, Karen Egiazarian, et al. 2014. Flexisp: A flexible camera image processing framework. ACM Transactions on Graphics (TOG) 33, 6 (2014), 231.
[34]
Minyoung Huh, Pulkit Agrawal, and Alexei A Efros. 2016. What makes ImageNet good for transfer learning? arXiv preprint arXiv:1608.08614 (2016).
[35]
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. 2017. Image-to-Image Translation with Conditional Adversarial Networks. CVPR (2017).
[36]
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. 2016. Perceptual losses for real-time style transfer and super-resolution. In European Conference on Computer Vision. Springer, 694--711.
[37]
Arnold Kalvach and Zsolt Szabó. 2016. Aberration-free flat lens design for a wide range of incident angles. Journal of the Optical Society of America B 33, 2 (2016), A66.
[38]
Rudolf Kingslake and R Barry Johnson. 2009. Lens design fundamentals. Academic Press.
[39]
Dilip Krishnan and Rob Fergus. 2009. Fast image deconvolution using hyper-Laplacian priors. In Advances in Neural Information Processing Systems. NIPS, 1033--1041.
[40]
Orest Kupyn, Volodymyr Budzan, Mykola Mykhailych, Dmytro Mishkin, and Jiri Matas. 2017. DeblurGAN: Blind Motion Deblurring Using Conditional Adversarial Networks. arXiv preprint arXiv:1711.07064 (2017).
[41]
Christian Ledig, Lucas Theis, Ferenc Huszár, Jose Caballero, Andrew Cunningham, Alejandro Acosta, Andrew P Aitken, Alykhan Tejani, Johannes Totz, Zehan Wang, et al. 2017. Photo-Realistic Single Image Super-Resolution Using a Generative Adversarial Network. In CVPR, Vol. 2. 4.
[42]
Anat Levin, Samuel W Hasinoff, Paul Green, Frédo Durand, and William T Freeman. 2009. 4D frequency analysis of computational cameras for depth of field extension. In ACM Transactions on Graphics (TOG), Vol. 28. ACM, 97.
[43]
Light.co. 2018. Light L16 Camera. https://light.co
[44]
Daniel Malacara-Hernández and Zacarías Malacara-Hernández. 2016. Handbook of optical design. CRC Press.
[45]
Kaushik Mitra, Oliver Cossairt, and Ashok Veeraraghavan. 2014. To denoise or deblur: parameter optimization for imaging systems. In Digital Photography X, Vol. 9023. International Society for Optics and Photonics, 90230G.
[46]
Anish Mittal, Anush Krishna Moorthy, and Alan Conrad Bovik. 2012. No-reference image quality assessment in the spatial domain. IEEE Transactions on Image Processing 21, 12 (2012), 4695--4708.
[47]
MobilEye. 2018. MobileEye TriCam. https://press.zf.com/site/press/en_de/microsites/press/list/release/release_41735.html
[48]
Mehjabin Monjur, Leonidas Spinoulas, Patrick R Gill, and David G Stork. 2015. Ultra-miniature, computationally efficient diffractive visual-bar-position sensor. In Proc. SensorComm. IEIFSA.
[49]
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. 2017. Deep multi-scale convolutional neural network for dynamic scene deblurring. In CVPR, Vol. 1. 3.
[50]
Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, Pat Hanrahan, et al. 2005. Light field photography with a hand-held plenoptic camera. (2005).
[51]
Yi-Ren Ng, Patrick M Hanrahan, Mark A Horowitz, and Marc S Levoy. 2012. Correction of optical aberrations. US Patent 8,243,157.
[52]
Augustus Odena, Vincent Dumoulin, and Chris Olah. 2016. Deconvolution and checkerboard artifacts. Distill 1, 10 (2016), e3.
[53]
Steve Oliver, Rick Lake, Shashikant Hegde, Jeff Viens, and Jacques Duparre. 2010. Imaging module with symmetrical lens system and method of manufacture. US Patent 7,710,667.
[54]
Marios Papas, Thomas Houit, Derek Nowrouzezahrai, Markus H Gross, and Wojciech Jarosz. 2012. The magic lens: refractive steganography. ACM Trans. Graph. 31, 6 (2012), 186--1.
[55]
Jim R Parker. 2010. Algorithms for image processing and computer vision. John Wiley & Sons.
[56]
Yifan Peng, Xiong Dun, Qilin Sun, and Wolfgang Heidrich. 2017. Mix-and-match holography. ACM Transactions on Graphics (TOG) 36, 6 (2017), 191.
[57]
Yifan Peng, Qiang Fu, Hadi Amata, Shuochen Su, Felix Heide, and Wolfgang Heidrich. 2015. Computational imaging using lightweight diffractive-refractive optics. Optics Express 23, 24 (2015), 31393--31407.
[58]
Yifan Peng, Qiang Fu, Felix Heide, and Wolfgang Heidrich. 2016. The diffractive achromat full spectrum computational imaging with diffractive optics. ACM Transactions on Graphics (TOG) 35, 4 (2016), 31.
[59]
Rajeev Ramanath, Wesley Snyder, Youngjun Yoo, and Mark Drew. 2005. Color image processing pipeline in digital still cameras. IEEE Signal Processing Magazine 22, 1 (2005), 34--43.
[60]
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. 2015. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer-assisted intervention. Springer, 234--241.
[61]
Christian J. Schuler, Harold Christopher Burger, Stefan Harmeling, and Bernhard Scholkopf. 2013. A Machine Learning Approach for Non-blind Image Deconvolution. In Proc. Computer Vision and Pattern Recognition.
[62]
Yuliy Schwartzburg, Romain Testuz, Andrea Tagliasacchi, and Mark Pauly. 2014. High-contrast computational caustic design. ACM Transactions on Graphics (TOG) 33, 4 (2014), 74.
[63]
Yichang Shih, Brian Guenter, and Neel Joshi. 2012. Image enhancement using calibrated lens simulations. In European Conference on Computer Vision. Springer, 42--56.
[64]
Vincent Sitzmann, Steven Diamond, Yifan Peng, Xiong Dun, Stephen Boyd, Wolfgang Heidrich, Felix Heide, and Gordon Wetzstein. 2018. End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging. ACM Transactions on Graphics (TOG) 37, 4 (2018), 114.
[65]
Georgii Georgievich Sliusarev. 1984. Aberration and optical design theory. Bristol, England, Adam Hilger, Ltd., 1984, 672 p. Translation. (1984).
[66]
Warren J. Smith. 2005. Modern lens design. McGraw-Hill.
[67]
David G Stork and Patrick R Gill. 2013. Lensless ultra-miniature CMOS computational imagers and sensors. Proc. SENSORCOMM (2013), 186--190.
[68]
David G Stork and Patrick R Gill. 2014. Optical, mathematical, and computational foundations of lensless ultra-miniature diffractive imagers and sensors. International Journal on Advances in Systems and Measurements 7, 3 (2014), 4.
[69]
Libin Sun, Sunghyun Cho, Jue Wang, and James Hays. 2013. Edge-based blur kernel estimation using patch priors. In Proc. International Conference on Computational Photography (ICCP). 1--8.
[70]
Kartik Venkataraman, Dan Lelescu, Jacques Duparré, Andrew McMahon, Gabriel Molina, Priyam Chatterjee, Robert Mullis, and Shree Nayar. 2013. Picam: An ultra-thin high performance monolithic camera array. ACM Transactions on Graphics (TOG) 32, 6 (2013), 166.
[71]
Peng Wang, Nabil Mohammad, and Rajesh Menon. 2016. Chromatic-aberration-corrected diffractive lenses for ultra-broadband focusing. Scientific Reports 6 (2016).
[72]
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. 2004. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing 13, 4 (2004), 600--612.
[73]
Yicheng Wu, Vivek Boominathan, Huaijin Chen, Aswin Sankaranarayanan, and Ashok Veeraraghavan. 2019. PhaseCam3D - Learning Phase Masks for Passive Single View Depth Estimation. In Proc. ICCP.
[74]
Li Xu, Jimmy SJ Ren, Ce Liu, and Jiaya Jia. 2014. Deep convolutional neural network for image deconvolution. In Advances in Neural Information Processing Systems. 1790--1798.
[75]
Xiaoyun Yuan, Lu Fang, Qionghai Dai, David J Brady, and Yebin Liu. 2017. Multiscale gigapixel video: A cross resolution image matching and warping approach. In 2017 IEEE International Conference on Computational Photography (ICCP). IEEE, 1--9.
[76]
Jiawei Zhang, Jinshan Pan, Wei-Sheng Lai, Rynson WH Lau, and Ming-Hsuan Yang. 2017. Learning fully convolutional networks for iterative non-blind deconvolution. (2017).
[77]
Zhengyou Zhang. 2000. A flexible new technique for camera calibration. IEEE Transactions on pattern analysis and machine intelligence 22 (2000).
[78]
Jun Zhu, Tong Yang, and Guofan Jin. 2013. Design method of surface contour for a freeform lens with wide linear field-of-view. Optics express 21, 22 (2013), 26080--26092.
[79]
Margarete Zoberbier, Sven Hansen, Marc Hennemeyer, Dietrich Tönnies, Ralph Zoberbier, Markus Brehm, Andreas Kraft, Martin Eisner, and Reinhard Völkel. 2009. Wafer level cameras---novel fabrication and packaging technologies. In Int. Image Sens. Workshop.

Cited By

View all
  • (2025)Representing domain-mixing optical degradation for real-world Computational Aberration Correction via vector quantizationOptics & Laser Technology10.1016/j.optlastec.2024.112080183(112080)Online publication date: May-2025
  • (2025)End-to-end aberration correction network for enhancing miniature microscope resolutionOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108558184(108558)Online publication date: Jan-2025
  • (2025)Image restoration for spatially varying degradation based on PSF propertiesOptics Communications10.1016/j.optcom.2024.131116574(131116)Online publication date: Jan-2025
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics
ACM Transactions on Graphics  Volume 38, Issue 6
December 2019
1292 pages
ISSN:0730-0301
EISSN:1557-7368
DOI:10.1145/3355089
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 November 2019
Published in TOG Volume 38, Issue 6

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. computational camera
  2. deep network
  3. image deblurring
  4. thin optics

Qualifiers

  • Research-article

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)181
  • Downloads (Last 6 weeks)18
Reflects downloads up to 12 Jan 2025

Other Metrics

Citations

Cited By

View all
  • (2025)Representing domain-mixing optical degradation for real-world Computational Aberration Correction via vector quantizationOptics & Laser Technology10.1016/j.optlastec.2024.112080183(112080)Online publication date: May-2025
  • (2025)End-to-end aberration correction network for enhancing miniature microscope resolutionOptics and Lasers in Engineering10.1016/j.optlaseng.2024.108558184(108558)Online publication date: Jan-2025
  • (2025)Image restoration for spatially varying degradation based on PSF propertiesOptics Communications10.1016/j.optcom.2024.131116574(131116)Online publication date: Jan-2025
  • (2024)中国光学十大进展:基于超构透镜阵列的新型成像器件(特邀)Laser & Optoelectronics Progress10.3788/LOP24074861:22(2200001)Online publication date: 2024
  • (2024)A thin and lightweight camera based on Pancake lens and deep learningOptics Letters10.1364/OL.531253Online publication date: 6-Aug-2024
  • (2024)End-to-end automatic lens design with a differentiable diffraction modelOptics Express10.1364/OE.54059032:25(44328)Online publication date: 20-Nov-2024
  • (2024)Configurable multiple virtual lenses conjugated with singlet physical lens for achromatic extended depth-of-field imagingOptics Express10.1364/OE.53867032:23(40427)Online publication date: 22-Oct-2024
  • (2024)Method for Optical Adjustment with Deep Learning to Quantitatively Predict Misalignment in OpticsApplied Optics10.1364/AO.532265Online publication date: 15-Aug-2024
  • (2024)4-K-resolution minimalist optical system design based on deep learningApplied Optics10.1364/AO.51086063:4(917)Online publication date: 22-Jan-2024
  • (2024)Simplified design method for optical imaging systems based on aberration characteristics of optical-digital joint optimizationApplied Optics10.1364/AO.51074663:4(1066)Online publication date: 25-Jan-2024
  • Show More Cited By

View Options

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media