article

Programmable Imaging: Towards a Flexible Camera

Authors:

Shree K. Nayar,

Terry E. BoultAuthors Info & Claims

International Journal of Computer Vision, Volume 70, Issue 1

Pages 7 - 22

https://doi.org/10.1007/s11263-005-3102-6

Published: 01 October 2006 Publication History

Abstract

In this paper, we introduce the notion of a programmable imaging system. Such an imaging system provides a human user or a vision system significant control over the radiometric and geometric characteristics of the system. This flexibility is achieved using a programmable array of micro-mirrors. The orientations of the mirrors of the array can be controlled with high precision over space and time. This enables the system to select and modulate rays from the scene's light field based on the needs of the application at hand.

We have implemented a programmable imaging system that uses a digital micro-mirror device (DMD), which is used in digital light processing. Although the mirrors of this device can only be positioned in one of two states, we show that our system can be used to implement a wide variety of imaging functions, including, high dynamic range imaging, feature detection, and object recognition. We also describe how a micro-mirror array that allows full control over the orientations of its mirrors can be used to instantly change the field of view and resolution characteristics of the imaging system. We conclude with a discussion on the implications of programmable imaging for computer vision.

References

[1]

Castracane, J. and Gutin, M. A. 1999. DMD-based bloom control for intensified imaging systems. In Diffractive and Holographic Tech., Syst., and Spatial Light Modulators VI, vol. 3633, pp. 234-242. SPIE.

[2]

Christensen, M.P., Euliss, G.W., McFadden, M.J., Coyle, K.M., Milojkovic, P., Haney, M.W., Gracht, J. van der, and Athale, R. A. October 2002. Active-eyes: An adaptive pixel-by-pixel image-segmentation sensor architecture for high-dynamic-range hyperspectral imaging. Applied Optics, 41(29):6093- 6103.

[3]

Dudley, D., Duncan, W., and Slaughter, J. February 2003. Emerging digital micromirror device (DMD) applications. White paper, Texas Instruments.

[4]

Ginosar, R., Hilsenrath, O., and Zeevi, Y. September 1992. Wide dynamic range camera. U.S. Patent 5, 144,442.

[5]

Goodman, J. W. 1968. Introduction to Fourier Optics. McGraw-Hill, New York.

[6]

Grossberg, M.D. and Nayar, S.K. 2005. The raxel imaging model and ray-based calibration. IJCV, 61 (2): 119-137.

Digital Library

[7]

Hicks, R. A. 2003. Personal Communication.

[8]

Hornbeck, L.J. 1988. Bistable deformable mirror device. In Spat. Light Mod. and Apps., vol. 8. OSA.

[9]

Hornbeck, L.J. August 1989. Deformable-mirror spatial light modulators. In Projection Displays III, vol. 1150, pp. 86-102. SPIE.

[10]

Hornbeck, L.J. 1995. Projection displays and MEMS: Timely convergence for a bright future. In Micromachined Devices and Components, vol. 2642. SPIE.

[11]

Kang, S.B., Uyttendaele, M., Winder, S., and Szeliski, R. 2003. High dynamic range video.ACM Trans. on Graph. (Proc. of SIGGRAPH 2003), 22(3):319-325.

[12]

Kearney, K.J. and Ninkov, Z. 1998. Characterization of digital micromirror device for use as an optical mask in imaging and spectroscopy. Spatial Light Modulators, 3292:81-92. SPIE.

[13]

Malbet, F., Yu, J., and Shao, M. 1995. High dynamic range imaging using a deformable mirror for space coronography. Public. of the Astro. Soc. of the Pacific, 107:386.

[14]

Murase, H. and Nayar, S.K. 1995. Visual learning and recognition of 3d objects from appearance. IJCV, 14(1):5-24.

Digital Library

[15]

Nayar, S.K. and Branzoi, V. 2003. Adaptive Dynamic Range Imaging: Optical Control of Pixel Exposures over Space and Time. Proc. of Inter. Conf. on Computer Vision (ICCV), pp. 1168- 1175.

[16]

Nayar, S.K. and Mitsunaga, T. 2000. High dynamic range imaging: Spatially varying pixel exposures. Proc. of IEEE Conf. on Computer Vision and Pattern Recognition, vol. 1:472- 479.

[17]

Nayar, S.K. and Narasimhan, S.G. 2002. Assorted pixels: Multisampled imaging with structural models. Proc. of Euro. Conf. on Comp. Vis. (ECCV), 4:636-152.

[18]

Smith, W.J. 1966. Modern Optical Engineering. McGraw-Hill.

[19]

Turk, M. and Pentland, A.P. 1991. Face recognition using eigenfaces. In Proc. of IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pp. 586-591.

[20]

Tyson, R.K. 1998. Principles of Adaptive Optics. Academic Press.

Cited By

Wang CZhang JWilson MEtienne-Cummings R(2024)Pix2HDR - A Pixel-Wise Acquisition and Deep Learning-Based Synthesis Approach for High-Speed HDR VideosIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.341014046:12(8771-8787)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3410140
Ducrocq JCaron G(2024)A Survey on Adaptive CamerasInternational Journal of Computer Vision10.1007/s11263-024-02025-7132:8(2989-3022)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s11263-024-02025-7
Tasneem ZAdhivarahan CWang DXie HDantu KKoppal S(2020)Adaptive fovea for scanning depth sensorsInternational Journal of Robotics Research10.1177/027836492092093139:7(837-855)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1177/0278364920920931
Show More Cited By

Index Terms

Programmable Imaging: Towards a Flexible Camera
1. Computer systems organization
  1. Embedded and cyber-physical systems
  2. Real-time systems
2. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Image and video acquisition
        Epipolar geometry
  2. Computer graphics
    1. Image manipulation

Recommendations

Multiview radial catadioptric imaging for scene capture
SIGGRAPH '06: ACM SIGGRAPH 2006 Papers

In this paper, we present a class of imaging systems, called radial imaging systems, that capture a scene from a large number of view-points within a single image, using a camera and a curved mirror. These systems can recover scene properties such as ...
Multiview radial catadioptric imaging for scene capture

In this paper, we present a class of imaging systems, called radial imaging systems, that capture a scene from a large number of view-points within a single image, using a camera and a curved mirror. These systems can recover scene properties such as ...
Flexible fabrication of large pixel count piston-tip-tilt mirror arrays for fast spatial light modulators

We present arrays of electrostatically actuated piston-tip-tilt micromirrors realized using a surface micromachining 3-structural-layer polysilicon process. High fill factor, flat mirrors 120@mm in size have mechanical response in the order of 10@ms, ...

Comments

Information & Contributors

Information

Published In

cover image International Journal of Computer Vision

International Journal of Computer Vision Volume 70, Issue 1

October 2006

99 pages

ISSN:0920-5691

Issue’s Table of Contents

Copyright © Copyright © 2006 Springer Science + Business Media, LLC.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 October 2006

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 08 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Wang CZhang JWilson MEtienne-Cummings R(2024)Pix2HDR - A Pixel-Wise Acquisition and Deep Learning-Based Synthesis Approach for High-Speed HDR VideosIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2024.341014046:12(8771-8787)Online publication date: 1-Dec-2024
https://dl.acm.org/doi/10.1109/TPAMI.2024.3410140
Ducrocq JCaron G(2024)A Survey on Adaptive CamerasInternational Journal of Computer Vision10.1007/s11263-024-02025-7132:8(2989-3022)Online publication date: 1-Aug-2024
https://dl.acm.org/doi/10.1007/s11263-024-02025-7
Tasneem ZAdhivarahan CWang DXie HDantu KKoppal S(2020)Adaptive fovea for scanning depth sensorsInternational Journal of Robotics Research10.1177/027836492092093139:7(837-855)Online publication date: 1-Jun-2020
https://dl.acm.org/doi/10.1177/0278364920920931
Khan SFeldman MGunturk B(2018)Extracting sub-exposure images from a single capture through Fourier-based optical modulationImage Communication10.1016/j.image.2017.09.01260:C(107-115)Online publication date: 1-Feb-2018
https://dl.acm.org/doi/10.1016/j.image.2017.09.012
Pittaluga FKoppal S(2017)Pre-Capture Privacy for Small Vision SensorsIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2016.263735439:11(2215-2226)Online publication date: 1-Nov-2017
https://dl.acm.org/doi/10.1109/TPAMI.2016.2637354
Berthaut FSahoo DMcIntosh JDas DSubramanian SNunes NCostanza EOlivier PSchöning J(2015)DynamirProceedings of the 2015 International Conference on Interactive Tabletops & Surfaces10.1145/2817721.2817744(55-58)Online publication date: 15-Nov-2015
https://dl.acm.org/doi/10.1145/2817721.2817744
Han SSato IOkabe TSato Y(2014)Fast Spectral Reflectance Recovery Using DLP ProjectorInternational Journal of Computer Vision10.1007/s11263-013-0687-z110:2(172-184)Online publication date: 1-Nov-2014
https://dl.acm.org/doi/10.1007/s11263-013-0687-z
Wetzstein GIhrke ILanman DHeidrich WRaskar RAkeley K(2012)Computational plenoptic imagingACM SIGGRAPH 2012 Courses10.1145/2343483.2343494(1-265)Online publication date: 5-Aug-2012
https://dl.acm.org/doi/10.1145/2343483.2343494
Sturm PRamalingam STardif JGasparini SBarreto J(2011)Camera Models and Fundamental Concepts Used in Geometric Computer VisionFoundations and Trends® in Computer Graphics and Vision10.1561/06000000236:1–2(1-183)Online publication date: 1-Jan-2011
https://dl.acm.org/doi/10.1561/0600000023
Han SSato IOkabe TSato Y(2010)Fast spectral reflectance recovery using DLP projectorProceedings of the 10th Asian conference on Computer vision - Volume Part I10.5555/1964320.1964353(323-335)Online publication date: 8-Nov-2010
https://dl.acm.org/doi/10.5555/1964320.1964353
Show More Cited By

View Options

View options

Figures

Tables

Media

View Issue’s Table of Contents