research-article

Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging

Authors:

Shree K. Nayar,

Matthias B. Hullin,

Jaime MartinAuthors Info & Claims

ACM Transactions on Graphics (TOG), Volume 34, Issue 5

Article No.: 156, Pages 1 - 18

https://doi.org/10.1145/2735702

Published: 03 November 2015 Publication History

Abstract

In correlation-based time-of-flight (C-ToF) imaging systems, light sources with temporally varying intensities illuminate the scene. Due to global illumination, the temporally varying radiance received at the sensor is a combination of light received along multiple paths. Recovering scene properties (e.g., scene depths) from the received radiance requires separating these contributions, which is challenging due to the complexity of global illumination and the additional temporal dimension of the radiance.

We propose phasor imaging, a framework for performing fast inverse light transport analysis using C-ToF sensors. Phasor imaging is based on the idea that, by representing light transport quantities as phasors and light transport events as phasor transformations, light transport analysis can be simplified in the temporal frequency domain. We study the effect of temporal illumination frequencies on light transport and show that, for a broad range of scenes, global radiance (inter-reflections and volumetric scattering) vanishes for frequencies higher than a scene-dependent threshold. We use this observation for developing two novel scene recovery techniques. First, we present micro-ToF imaging, a ToF-based shape recovery technique that is robust to errors due to inter-reflections (multipath interference) and volumetric scattering. Second, we present a technique for separating the direct and global components of radiance. Both techniques require capturing as few as 3--4 images and minimal computations. We demonstrate the validity of the presented techniques via simulations and experiments performed with our hardware prototype.

Supplementary Material

a156-gupta-app.pdf (gupta.zip)

Supplemental movie, appendix, image and software files for, Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging

Download
207.49 KB

MP4 File (a156.mp4)

Download
14.83 MB

References

[1]

X. Ai, R. Nock, J. G. Rarity, and N. Dahnoun. 2011. High-resolution random-modulation CW lidar. Appl. Optics 50, 22.

[2]

M. Akbulut, C. H. Chen, M. C. Hargis, A. M. Weiner, M. R. Mel-Loch, and J. M. Woodall. 2001. Digital communications above 1 Gb/s using 890-nm surface-emitting light-emitting diodes. IEEE Photon. Technol. Lett. 13, 1.

[3]

J. Busck and H. Heiselberg. 2004. High accuracy 3D laser radar. Proc. SPIE 5412.

[4]

B. Buttgen, M.-A. El Mechat, F. Lustenberger, and P. Seitz. 2007. Pseudonoise optical modulation for real-time 3-d imaging with minimum interference. IEEE Trans. Circ. Syst. 54, 10, 2109--2119.

[5]

B. Buxbaum, R. Schwarte, T. Ringbeck, M. Grothof, and X. Luan. 2002. Msm-pmd as correlation receiver in a new 3d-ranging system. Proc. SPIE 4546.

[6]

D. A. Carnegie, J. R. K. McClymont, A. P. P. Jongenelen, B. Drayto, A. A. Dorrington, and A. D. Payne. 2011. Design and construction of a configurable full-field range imaging system for mobile robotic applications. In New Developments and Applications in Sensing Technology. Springer, 133--155.

[7]

C. H. Chen, M. Hargis, J. M. Woodall, M. R. Melloch, J. S. Reynolds, E. Yablonovitch, and W. Wang. 1999. Ghz bandwidth gaas light-emitting diodes. Appl. Phys. Lett. 74, 21.

[8]

T. Chen, H. P. Seidel, and H. Lensch. 2008. Modulated phase-shifting for 3D scanning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'08). 1--8.

[9]

V. Couture, N. Martin, and S. Roy. 2014. Unstructured light scanning robust to indirect illumination and depth discontinuities. Int. J. Comput. Vis. 108, 3, 204--221.

Digital Library

[10]

A. A. Dorrington, J. P. Godbaz, M. J. Cree, A. D. Payne, and L. V. Streeter. 2011. Separating true range measurements from multi-path and scattering interference in commercial range cameras. Proc. SPIE 7864.

[11]

R. Ferriere, J. Cussey, and J. Dudley. 2008. Time-of-flight range detection using low-frequency intensity modulation of a CW laser diode: Application to fiber length measurement. Optical Engin. 47, 9.

[12]

D. Freedman, E. Krupka, Y. Smolin, I. Leichter, and M. Schmidt. 2014. SRA: Fast removal of general multipath for ToF sensors. In Proceedings of the 13^th European Conference on Computer Vision (ECCV'14). 234--249.

[13]

S. Fuchs. 2010. Multipath interference compensation in time-of-flight camera images. In Proceedings of the 20^th International Conference on Pattern Recognition (ICPR'10). 3583--3586.

Digital Library

[14]

S. Fuchs, M. Suppa, and O. Hellwich. 2013. Compensation for multipath in Tof camera measurements supported by photometric calibration and environment integration. In Proceedings of the 9^th International Conference on Computer Vision Systems (ICVS'13). 31--41.

Digital Library

[15]

J. Godbaz, M. Cree, and A. Dorrington. 2008. Mixed pixel return separation for a full-field ranger. In Proceedings of the 23^rd International Conference on Image and Vision Computing New Zealand (IVCNZ'08). 1--6.

[16]

J. P. Godbaz, M. J. Cree, and A. A. Dorrington. 2009. Multiple return separation for a full-field ranger via continuous waveform modelling. Proc. SPIE 7251.

[17]

J. P. Godbaz, M. J. Cree, and A. A. Dorrington. 2012. Closed-form inverses for the mixed pixel/multipath interference problem in AMCW lidar. Proc. SPIE 8296.

[18]

S. B. Gokturk, H. Yalcin, and C. Bamji. 2004. A time-of-flight depth sensor - System description, issues and solutions. In Proceedings of the Conference on Computer Vision and Pattern Recognition Workshop (CVPRW'04). 35.

Digital Library

[19]

J. Gu, T. Kobayashi, M. Gupta, and S. Nayar. 2011. Multiplexed illumination for scene recovery in the presence of global illumination. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'11). 691--698.

Digital Library

[20]

M. Gupta, A. Agrawal, A. Veeraraghavan, and S. G. Narasimhan. 2013a. A practical approach to 3d scanning in the presence of interreflections, subsurface scattering and defocus. Int. J. Comput. Vis. 102, 1--3, 33--55.

Digital Library

[21]

M. Gupta, Q. Yin, and S. K. Nayar. 2013b. Structured light in sunlight. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'13). 545--552.

Digital Library

[22]

M. Gupta and S. K. Nayar. 2012. Micro phase shifting. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 813--820.

Digital Library

[23]

M. Gupta, Y. Tian, S. G. Narasimhan, and L. Zhang. 2009. (De)focusing on global light transport for active scene recovery. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'09). 2969--2976.

[24]

V. I. Gushov and Y. N. Solodkin. 1991. Automatic processing of fringe patterns in integer interferometers. Optics Lasers Engin. 14, 4--5.

[25]

S. W. Hasinoff, F. Durand, and W. T. Freeman. 2010. Noise-optimal capture for high dynamic range photography. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'10). 553--560.

[26]

F. Heide, M. B. Hullin, J. Gregson, and W. Heidrich. 2013. Low-budget transient imaging using photonic mixer devices. ACM Trans. Graph. 32, 4.

Digital Library

[27]

J. Heinen, W. Huber, and W. Harth. 1976. Light-emitting diodes with a modulation bandwidth of more than 1 Ghz. Electron. Lett. 12, 21.

[28]

L. G. Henyey and J. L. Greenstein. 1941. Diffuse radiation in the galaxy. Astrophys. J. 93, 70--83.

[29]

D. Jimenez, D. Pizarro, M. Mazo, and S. Palazuelos. 2012. Modelling and correction of multipath interference in time of flight cameras. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 893--900.

Digital Library

[30]

A. P. P. Jongenelen, D. G. Bailey, A. D. Payne, A. A. Dorrington, and D. A. Carnegie. 2011. Analysis of errors in tof range imaging with dual-frequency modulation. IEEE Trans. Instrument. Measure. 60, 5.

[31]

A. P. P. Jongenelen, D. Carnegie, A. D. Payne, and A. A. Dorrington. 2010. Maximizing precision over extended unambiguous range for Tof range imaging systems. In Proceedings of the IEEE Instrumentation and Measurement Technology Conference (I2MTC'10). 1575--1580.

[32]

A. Kadambi, R. Whyte, A. Bhandari, L. Streeter, C. Barsi, A. Dorrington, and R. Raskar. 2013. Coded time of flight cameras: Sparse deconvolution to address multipath interference and recover time profiles. ACM Trans. Graph. 32, 6.

Digital Library

[33]

A. Kirmani, A. Benedetti, and P. Chou. 2013. SPUMIC: Simultaneous phase unwrapping and multipath interference cancellation in time-of-flight cameras using spectral methods. In Proceedings of the IEEE International Conference on Multimedia and Expo (ICME'13). 1--6.

[34]

A. Kirmani, T. Hutchison, J. Davis, and R. Raskar. 2009. Looking around the corner using transient imaging. In Proceedings of the IEEE International Conference on Computer Vision (ICCV'09). 159--166.

[35]

W. Koechner. 1968. Optical ranging system employing a high power injection laser diode. IEEE Trans. Aerospace Electron. Syst. 4, 1.

[36]

R. Lange. 2000. 3d time-of-flight distance measurement with custom solid-state image sensors in cmos-ccd-technology. PhD thesis. http://d-nb.info/960293825/34.

[37]

R. Lange and P. Seitz. 2001. Solid state time-of-flight range camera. IEEE J. Quantum Electron. 37, 3.

[38]

N. Naik, S. Zhao, A. Velten, R. Raskar, and K. Bala. 2011. Single view reflectance capture using multiplexed scattering and time-of-flight imaging. ACM Trans. Graph. 30, 6.

Digital Library

[39]

S. G. Narasimhan, M. Gupta, C. Donner, R. Ramamoorthi, S. K. Nayar, and H. W. Jensen. 2006. Acquiring scattering properties of participating media by dilution. ACM Trans. Graph. 25, 3, 1003--1012.

Digital Library

[40]

S. K. Nayar, G. Krishnan, M. D. Grossberg, and R. Raskar. 2006. Fast separation of direct and global components of a scene using high frequency illumination. ACM Trans. Graph. 25, 3, 935--944.

Digital Library

[41]

M. O'Toole, F. Heide, L. Xiao, M. B. Hullin, W. Heidrich, and K. N.Kutulakos. 2014. Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans. Graph. 33, 4.

Digital Library

[42]

M. O'Toole, R. Raskar, and K. N. Kutulakos. 2012. Primal-dual coding to probe light transport. ACM Trans. Graph. 31, 4.

Digital Library

[43]

R. Pandharkar, A. Velten, A. Bardagjy, E. Lawson, M. Bawendi, and R. Raskar. 2011. Estimating motion and size of moving non-line-of-sight objects in cluttered environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'11).265--272.

Digital Library

[44]

A. D. Payne, A. A. Dorrington, and M. J. Cree. 2010a. Illumination waveform optimization for time-of-flight range imaging cameras. Proc. SPIE 8085.

[45]

A. D. Payne, A. A. Dorrington, M. J. Cree, and D. A. Carnegie. 2010b. Improved measurement linearity and precision for AMCW time-of-flight range imaging cameras. Appl. Optics 49, 23.

[46]

D. Reddy, R. Ramamoorthi, and B. Curless. 2012. Frequency-space decomposition and acquisition of light transport under spatially varying illumination. In Proceedings of the 12^th European Conference on Computer Vision (ECCV'12). 596--610.

Digital Library

[47]

R. Schwarte. 2004. Breakthrough in multi-channel laser-radar technology providing thousands of high-sensitive lidar receivers on a chip. Proc. SPIE 5575.

[48]

R. Schwarte, Z. Xu, H. Heinol, J. Olk, R. Klein, B. Buxbaum, H. Fischer, and J. Schulte. 1997. New electro-optical mixing and correlating sensor: Facilities and applications of the photonic mixer device. Proc. SPIE 3100.

[49]

S. M. Seitz, Y. Matsushita, and K. N. Kutulakos. 2005. A theory of inverse light transport. In Proceedings of the 10^th IEEE International Conference on Computer Vision (ICCV'05). 1440--1447.

Digital Library

[50]

M. Takeda, Q. Gu, M. Kinoshita, H. Takai, and Y. Takahashi. 1997. Frequency-multiplex fourier-transform profilometry: A single-shot three-dimensional shape measurement of objects with large height discontinuities and/or surface isolations. Appl. Optics 36, 22.

[51]

A. Velten, T. Willwacher, O. Gupta, A. Veeraraghavan, M. G. Bawendi, and R. Raskar. 2012. Recovering three-dimensional shape around a corner using ultrafast time-of-flight imaging. Nature 3, 745.

[52]

A. Velten, D. Wu, A. Jarabo, B. Masia, C. Barsi, C. Joshi, E. Lawson, M. Bawendi, D. Gutierrez, and R. Raskar. 2013. Femto-photography: Capturing and visualizing the propagation of light. ACM Trans. Graph. 32, 4.

Digital Library

[53]

G. Walter, C. Wu, H. Then, M. Feng, and N. Holonyak. 2009. Tilted-charge high speed (7 Ghz) light emitting diode. Appl. Phys. Lett. 94, 23.

[54]

C. H. Wu, G. Walter, H. Then, and M. Feng. 2010. Design and layout of multi Ghz operation of light emitting diodes. In Proceedings of the GaAs MAN-TECH Conference.

[55]

D. Wu, M. O'Toole, A. Velten, A. Agrawal, and R. Raskar. 2012a. Decomposing global light transport using time of flight imaging. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR'12). 366--373.

Digital Library

[56]

D. Wu, G. Wetzstein, C. Barsi, T. Willwacher, M. O'Toole, N. Naik, Q. Dai, K. Kutulakos, and R. Raskar. 2012b. Frequency analysis of transient light transport with applications in bare sensor imaging. In Proceedings of the 12^th European Conference on Computer Vision (ECCV'12).542--555.

Digital Library

Cited By

(Hsin) Pan CWalsh NZhang YShi ZHeide F(2024)Neural single-shot GHz FMCW correlation imagingOptics Express10.1364/OE.51903132:16(27835)Online publication date: 18-Jul-2024
https://doi.org/10.1364/OE.519031
Folden JAlley DIllig DMullen LKoppal S(2024)Confocal Bistatic LIDAR in scattering mediaOcean Sensing and Monitoring XVI10.1117/12.3014721(8)Online publication date: 6-Jun-2024
https://doi.org/10.1117/12.3014721
Zhang YLau D(2024)BimodalPS: Causes and Corrections for Bimodal Multi-Path in Phase-Shifting Structured Light ScannersIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.320626546:6(4001-4017)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2022.3206265
Show More Cited By

Index Terms

Phasor Imaging: A Generalization of Correlation-Based Time-of-Flight Imaging
1. Computing methodologies
  1. Artificial intelligence
    1. Computer vision
      1. Computer vision problems
        Shape inference
  2. Computer graphics
    1. Shape modeling

Recommendations

Shield fields: modeling and capturing 3D occluders

We describe a unified representation of occluders in light transport and photography using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. Our key theoretical result is that shield fields can be used to ...
Computational Schlieren Photography with Light Field Probes

We introduce a new approach to capturing refraction in transparent media, which we call light field background oriented Schlieren photography. By optically coding the locations and directions of light rays emerging from a light field probe, we can ...
Shield fields: modeling and capturing 3D occluders
SIGGRAPH Asia '08: ACM SIGGRAPH Asia 2008 papers

We describe a unified representation of occluders in light transport and photography using shield fields: the 4D attenuation function which acts on any light field incident on an occluder. Our key theoretical result is that shield fields can be used to ...

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Graphics

ACM Transactions on Graphics Volume 34, Issue 5

October 2015

188 pages

ISSN:0730-0301

EISSN:1557-7368

DOI:10.1145/2843519

Editor:
Kavita Bala
Cornell University

Issue’s Table of Contents

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 03 November 2015

Accepted: 01 February 2015

Revised: 01 December 2014

Received: 01 June 2014

Published in TOG Volume 34, Issue 5

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed

Funding Sources

NSF

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

120
Total Citations
View Citations
1,151
Total Downloads

Downloads (Last 12 months)135
Downloads (Last 6 weeks)10

Reflects downloads up to 20 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

(Hsin) Pan CWalsh NZhang YShi ZHeide F(2024)Neural single-shot GHz FMCW correlation imagingOptics Express10.1364/OE.51903132:16(27835)Online publication date: 18-Jul-2024
https://doi.org/10.1364/OE.519031
Folden JAlley DIllig DMullen LKoppal S(2024)Confocal Bistatic LIDAR in scattering mediaOcean Sensing and Monitoring XVI10.1117/12.3014721(8)Online publication date: 6-Jun-2024
https://doi.org/10.1117/12.3014721
Zhang YLau D(2024)BimodalPS: Causes and Corrections for Bimodal Multi-Path in Phase-Shifting Structured Light ScannersIEEE Transactions on Pattern Analysis and Machine Intelligence10.1109/TPAMI.2022.320626546:6(4001-4017)Online publication date: Jun-2024
https://doi.org/10.1109/TPAMI.2022.3206265
Dong GZhang YSun XXiong Z(2024)Recurrent Cross-Modality Fusion for Time-of-Flight Depth DenoisingIEEE Transactions on Computational Imaging10.1109/TCI.2024.349631210(1626-1639)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3496312
Wang TWang JMatsuda NCossairt OWillomitzer F(2024)Differentiable Deflectometric Eye TrackingIEEE Transactions on Computational Imaging10.1109/TCI.2024.338249410(888-898)Online publication date: 2024
https://doi.org/10.1109/TCI.2024.3382494
Shahandashti PLópez PBrea VGarcía-Lesta DConde M(2024)Simultaneous Multifrequency Demodulation for Single-Shot Multiple-Path ToF ImagingIEEE Transactions on Computational Imaging10.1109/TCI.2023.334875810(54-68)Online publication date: 2024
https://doi.org/10.1109/TCI.2023.3348758
Luo WIwaguchi TSagawa RKawasaki H(2024)Multi-Path Interference Mitigation For Indirect Time-of-Flight Camera By the Distortion of Coding Curve2024 IEEE International Conference on Image Processing (ICIP)10.1109/ICIP51287.2024.10648103(2779-2785)Online publication date: 27-Oct-2024
https://doi.org/10.1109/ICIP51287.2024.10648103
Paredes AGutierrez-Barragan FConde M(2024)Coherence, Distance and Error: Understanding Coded Demodulation in PB-ToF Imaging2024 International Workshop on the Theory of Computational Sensing and its Applications to Radar, Multimodal Sensing and Imaging (CoSeRa)10.1109/CoSeRa60846.2024.10720373(112-116)Online publication date: 18-Sep-2024
https://doi.org/10.1109/CoSeRa60846.2024.10720373
Friday SShi YCherivirala YSaragadam VPediredla A(2024)Snapshot Lidar: Fourier Embedding of Amplitude and Phase for Single-Image Depth Reconstruction2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.02381(25203-25212)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.02381
Zhou XDuan PLi BZhou CXu CShi B(2024)EvDiG: Event-guided Direct and Global Components Separation2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)10.1109/CVPR52733.2024.00918(9612-9621)Online publication date: 16-Jun-2024
https://doi.org/10.1109/CVPR52733.2024.00918
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents