article

Projection defocus analysis for scene capture and image display

Authors:

Shree NayarAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 25, Issue 3

Pages 907 - 915

https://doi.org/10.1145/1141911.1141974

Published: 01 July 2006 Publication History

Abstract

In order to produce bright images, projectors have large apertures and hence narrow depths of field. In this paper, we present methods for robust scene capture and enhanced image display based on projection defocus analysis. We model a projector's defocus using a linear system. This model is used to develop a novel temporal defocus analysis method to recover depth at each camera pixel by estimating the parameters of its projection defocus kemel in frequency domain. Compared to most depth recovery methods, our approach is more accurate near depth discontinuities. Furthermore, by using a coaxial projector-camera system, we ensure that depth is computed at all camera pixels, without any missing parts. We show that the recovered scene geometry can be used for refocus synthesis and for depth-based image composition. Using the same projector defocus model and estimation technique, we also propose a defocus compensation method that filters a projection image in a spatially-varying, depth-dependent manner to minimize its defocus blur after it is projected onto the scene. This method effectively increases the depth of field of a projector without modifying its optics. Finally, we present an algorithm that exploits projector defocus to reduce the strong pixelation artifacts produced by digital projectors, while preserving the quality of the projected image. We have experimentally verified each of our methods using real scenes.

Supplementary Material

JPG File (p907-zhang-high.jpg)

Download
9.24 KB

JPG File (p907-zhang-low.jpg)

Download
7.91 KB

High Resolution (p907-zhang-high.mov)

Download
35.49 MB

Low Resolution (p907-zhang-low.mov)

Download
13.99 MB

References

[1]

Bimber, O., and Emmerling, A. 2006. Multi-focal projection. IEEE Trans. on Visualization and Computer Graphics to appear.

[2]

Bimber, O., Wetzstein, G., Emmerling, A., and Nitschke, C. 2005. Enabling view-dependent stereoscopic projection in real environments. In Proc. Int. Symp. on Mixed and Augmented Reality, 14--23.

Digital Library

[3]

Curless, B., and Levoy, M. 1995. Better optical triangulation through spacetime analysis. In Proc. Int. Conf. on Computer Vision, 987--994.

Digital Library

[4]

Davis, J., Nehab, D., Ramamoothi, R., and Rusinkiewicz, S. 2005. Space-time stereo: A unifying framework for depth from triangulation. IEEE Trans. on Pattern Analysis and Machine Intelligence 27, 2, 296--302.

Digital Library

[5]

Faugeras, O. 1993. Three-Dimensional Computer Vision. MIT Press.

Digital Library

[6]

Favaro, P., and Soatto, S. 2005. A geometric approach to shape from defocus. IEEE Trans. on Pattern Analysis and Machine Intelligence (in press).

Digital Library

[7]

Fujii, K., Grossberg, M., and Nayar, S. 2005. A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 814--821.

Digital Library

[8]

Girod, B., and Scherock, S. 1989. Depth from defocus of structured light. In Proc. SPIE Conf. on Optics, Illumination, and Image Sensing for Machine Vision.

[9]

Gonzales-Banos, H., and Davis, J. 2004. A method for computing depth under ambient illumination using multi-shuttered light. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 234--241.

Digital Library

[10]

Grossberg, M., Peri, H., Nayar, S., and Belhumeur, P. 2004. Making One Object Look Like Another: Controlling Appearance using a Projector-Camera System. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), vol. I, 452--459.

[11]

Horn, B., and Brooks, M. 1989. Shape from Shading. MIT Press.

Digital Library

[12]

Huang, P. S., Zhang, C. P., and Chiang, F. P. 2003. High speed 3-d shape measurement based on digital fringe projection. Optical Engineering 42, 1, 163--168.

[13]

Jain, A. K. 1989. Fundamentals of Digital Image Processing. Prentice Hall.

Digital Library

[14]

Jin, H., and Favaro, P. 2002. A variational approach to shape from defocus. In Proc. Eur. Conf. on Computer Vision, 18--30.

Digital Library

[15]

Kanade, T., Gruss, A., and Carley, L. 1991. A very fast vlsi rangefinder. In Proc. Int. Conf. on Robotics and Automation, vol. 39, 1322--1329.

[16]

Koninckx, T. P., Peers, P., Dutr, P., and Gool, L. V. 2005. Scene-adapted structured light. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 611--619.

Digital Library

[17]

Levoy, M., Chen, B., Vaish, V., Horowitz, M., Mcdowall, I., and Bolas, M. 2004. Synthetic aperture confocal imaging. In SIGGRAPH Conference Proceedings, 825--834.

Digital Library

[18]

Ljung, L. 1998. System Identification: A Theory for the User, 2 ed. Prentice Hall.

Digital Library

[19]

Majumder, A., and Welch, G. 2001. Computer graphics optique: Optical superposition of projected computer graphics. In Proc. Eurographics Workshop on Virtual Enviroment/Immersive Projection Technology.

Digital Library

[20]

Mcguire, M., Matusik, W., Pfister, H., Hughes, J. F., and Durand, F. 2005. Defocus video matting. In SIGGRAPH Conference Proceedings, 567--576.

Digital Library

[21]

Nayar, S. K., and Nakagawa, Y. 1994. Shape from focus. IEEE Trans. on Pattern Analysis and Machine Intelligence 16, 8, 824--831.

Digital Library

[22]

Nayar, S. K., Watanabe, M., and Noguchi, M. 1996. Real-time focus range sensor. IEEE Transactions on Pattern Analysis and Machine Intelligence 18, 12, 1186--1198.

Digital Library

[23]

Nocedal, J., and Wright, S. J. 1999. Numerical Optimization. Springer.

[24]

Oppenheim, A. V., and Willsky, A. S. 1997. Signals and Systems, 2 ed. Prentice Hall.

Digital Library

[25]

Pentland, A. 1987. A new sense for depth of field. IEEE Trans. on Pattern Analysis and Machine Intelligence 9, 4, 523--531.

Digital Library

[26]

Raj, A., and Zabih, R. 2005. A graph cut algorithm for generalized image deconvolution. In Proc. Int. Conf. on Computer Vision.

Digital Library

[27]

Rajagopalan, A. N., and Chaudhuri, S. 1997. A variational approach to recovering depth from defocused images. IEEE Trans. on Pattern Analysis and Machine Intelligence 19, 10, 1158--1164.

Digital Library

[28]

Raskar, R., Welch, G., Cutts, M., Lake, A., Stesin, L., and Fuchs, H. 1998. The office of the future: A unified approach to image-based modeling and spatially immersive displays. In SIGGRAPH Conference Proceedings, 179--188.

Digital Library

[29]

Raskar, R., Welch, G., Low, K., and Bandyopadhyay, D. 2001. Shader lamps. In Proc. Eurographics Workshop on Rendering.

[30]

Raskar, R., Van Baar, J., Beardsley, P., Willwacher, T., Rao, S., and Forlines, C. 2003. ilamps: geometrically aware and self-configuring projectors. In SIGGRAPH Conference Proceedings, 809--818.

Digital Library

[31]

Raskar, R., Han Tan, K., Feris, R., Yu, J., and Turk, M. 2004. Non-photorealistic camera: Depth edge detection and stylized rendering using multi-flash imaging. In SIGGRAPH Conference Proceedings, 679--688.

Digital Library

[32]

Scharstein, D., and Szeliski, R. 2003. High-accuracy stereo depth maps using structured light. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, 195--202.

Digital Library

[33]

Schechner, Y. Y., Kiryati, N., and Basri, R. 2000. Separation of transparent layers using focus. Int. J. on Computer Vision 39, 1, 25--39.

Digital Library

[34]

Sen, P., Chen, B., Garg, G., Marschner, S. R., Horowitz, M., Levoy, M., and Lensch, H. P. A. 2005. Dual photography. In SIGGRAPH Conference Proceedings, 745--755.

Digital Library

[35]

Tappen, M. F., Russell, B. C., and Freeman, W. T. 2004. Efficient graphical models for processing images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition, vol. 2, 673--680.

Digital Library

[36]

Zhang, L., Snavely, N., Curless, B., and Seitz, S. M. 2004. Spacetime faces: High-resolution capture for modeling and animation. In ACM Annual Conference on Computer Graphics, 548--558.

Digital Library

[37]

Zhang, Z. 2000. A flexible new technique for camera calibration. IEEE Trans. on Pattern Analysis and Machine Intelligence 22, 11, 1330--1334.

Digital Library

Cited By

Yao PChen YGai SDa F(2024)Accurate 3D Measurement of Complex Texture Objects by Height Compensation Using a Dual-Projector StructureIEEE Transactions on Image Processing10.1109/TIP.2024.338960933(3021-3030)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3389609
Lyu NZheng DBai LYu HHan J(2024)Single-pixel imaging-based PSF compensation for large depth-of-field fringe projection profilometryMeasurement10.1016/j.measurement.2024.114954235(114954)Online publication date: Aug-2024
https://doi.org/10.1016/j.measurement.2024.114954
Cai WZhang XLi HLiao Z(2023)Depth of Field Extension for Microscopic Structured Light Vision System Based on Image Fusion2023 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS)10.1109/MARSS58567.2023.10294162(1-6)Online publication date: 9-Oct-2023
https://doi.org/10.1109/MARSS58567.2023.10294162
Show More Cited By

Index Terms

Projection defocus analysis for scene capture and image display
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
      2. Computer vision tasks
        Scene understanding
  2. Computer graphics
    1. Image manipulation
    2. Rendering

Recommendations

Projection defocus analysis for scene capture and image display
SIGGRAPH '06: ACM SIGGRAPH 2006 Papers

In order to produce bright images, projectors have large apertures and hence narrow depths of field. In this paper, we present methods for robust scene capture and enhanced image display based on projection defocus analysis. We model a projector's ...
Multi-epipolar plane image based 3D reconstruction using robust surface fitting
ICVGIP '14: Proceedings of the 2014 Indian Conference on Computer Vision Graphics and Image Processing

In this paper, we present a novel method for 3D reconstruction from epipolar plane (EP) representation of images and surface fitting for multiview or lightfield images. The proposed method detects parallelograms in EP images using mean shift ...
Optimal depth recovery using image guided TGV with depth confidence for high-quality view synthesis

A confidence-based depth recovery and high quality 3D view synthesis are proposed.The depth recovery relies on image edges and high depth confidence pixels.Texture directions of background are effective in hole filling for view synthesis. This paper ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 25, Issue 3

July 2006

742 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/1141911

Issue’s Table of Contents

Copyright © 2006 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 July 2006

Published in TOG Volume 25, Issue 3

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

87
Total Citations
View Citations
1,991
Total Downloads

Downloads (Last 12 months)11
Downloads (Last 6 weeks)1

Reflects downloads up to 29 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Yao PChen YGai SDa F(2024)Accurate 3D Measurement of Complex Texture Objects by Height Compensation Using a Dual-Projector StructureIEEE Transactions on Image Processing10.1109/TIP.2024.338960933(3021-3030)Online publication date: 22-Apr-2024
https://dl.acm.org/doi/10.1109/TIP.2024.3389609
Lyu NZheng DBai LYu HHan J(2024)Single-pixel imaging-based PSF compensation for large depth-of-field fringe projection profilometryMeasurement10.1016/j.measurement.2024.114954235(114954)Online publication date: Aug-2024
https://doi.org/10.1016/j.measurement.2024.114954
Cai WZhang XLi HLiao Z(2023)Depth of Field Extension for Microscopic Structured Light Vision System Based on Image Fusion2023 International Conference on Manipulation, Automation and Robotics at Small Scales (MARSS)10.1109/MARSS58567.2023.10294162(1-6)Online publication date: 9-Oct-2023
https://doi.org/10.1109/MARSS58567.2023.10294162
Ueda FKageyama YIwai DSato K(2023)Focal surface projection: Extending projector depth of field using a phase‐only spatial light modulatorJournal of the Society for Information Display10.1002/jsid.126131:11(651-656)Online publication date: 3-Oct-2023
https://doi.org/10.1002/jsid.1261
Akiyama RFukiage TNishida S(2022)Perceptually-Based Optimization for Radiometric Projector Compensation2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW)10.1109/VRW55335.2022.00226(750-751)Online publication date: Mar-2022
https://doi.org/10.1109/VRW55335.2022.00226
Huang BSun TLing H(2022)End-to-End Full Projector CompensationIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2021.305012444:6(2953-2967)Online publication date: 1-Jun-2022
https://doi.org/10.1109/TPAMI.2021.3050124
Wang XZhu DShi WLiu JFu FLi JZhang X(2022)Multi-Depth-of-Field 3-D Profilometry for a Microscopic System With Telecentric LensIEEE Transactions on Instrumentation and Measurement10.1109/TIM.2021.313715671(1-9)Online publication date: 2022
https://doi.org/10.1109/TIM.2021.3137156
Long QLi HYang MYang X(2022)An Improved Time Defocus Analysis Method for Measuring 3D2022 IEEE 6th Advanced Information Technology, Electronic and Automation Control Conference (IAEAC )10.1109/IAEAC54830.2022.9929976(1937-1940)Online publication date: 3-Oct-2022
https://doi.org/10.1109/IAEAC54830.2022.9929976
Barzi FNezamabadi-pour H(2022)Automatic objects’ depth estimation based on integral imagingMultimedia Tools and Applications10.1007/s11042-022-13221-381:30(43531-43549)Online publication date: 1-Dec-2022
https://dl.acm.org/doi/10.1007/s11042-022-13221-3
Wang YZhao HJiang HLi XLi YXu Y(2021)Paraxial 3D shape measurement using parallel single-pixel imagingOptics Express10.1364/OE.43547029:19(30543)Online publication date: 7-Sep-2021
https://doi.org/10.1364/OE.435470
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Issue’s Table of Contents