Abstract
Single-photon avalanche diodes (SPADs) are growing in popularity for depth sensing tasks. However, SPADs still struggle in the presence of high ambient light due to the effects of pile-up. Conventional techniques leverage fixed or asynchronous gating to minimize pile-up effects, but these gating schemes are all non-adaptive, as they are unable to incorporate factors such as scene priors and previous photon detections into their gating strategy. We propose an adaptive gating scheme built upon Thompson sampling. Adaptive gating periodically updates the gate position based on prior photon observations in order to minimize depth errors. Our experiments show that our gating strategy results in significantly reduced depth reconstruction error and acquisition time, even when operating outdoors under strong sunlight conditions.
SPAD 3D imaging setup
Captured results
While conventional methods for pile-up compensation employ either a static gate or uniformly distributed gate (asynchronous acquisition/free running mode). The former requires a strong prior on scene depth to observe reasonable improvements in depth reconstruction errors while the latter tends to fail under high ambient light and low integration times. We propose an adaptive method for hardware gating, where gate positions are selected depending on prior information obtained from previous photon observations.
Conventional approach:
static gating
Conventional approach:
asynchronous gating
Our approach:
adaptive gating
The core of our method relies on a well established algorithm in the field of multi-armed bandit problems known as Thompson sampling. Thompson sampling bridges the gap between exploration and exploitation by obtaining estimates of hidden variable through sampling the posterior distribution. For our adaptive gating algorithm, we employ a pulse-by-pulse adaptive procedure. For every SPAD cycle we do the following: 1. we sample the current depth distribution to obtain a proxy for true depth, 2. we set the gate as the sampled depth, 3. we wait for a photon detection, and 4. update our running depth distribution. We then obtain the final depth estimate from the resulting depth distribution at the end of acquisition.
Adaptive Exposure
Through the adaptive gating framework, we can reduce total acquisition time by stopping SPAD acquisition when the depth distribution has already converged, we call this method adaptive exposure. Note that time for convergence greatly depends on background illumination of the scene point, meaning certain points in the scene may take longer integration times to obtain accurate estimates. Adaptive exposure assigns varying integration times for each scene point, making sure errors aren’t concentrated in certain parts of a scene and reducing total acquisition time while maintaining equal or better reconstruction accuracy.
Up-stream Compatibility
In our adaptive gating algorithm, without prior knowledge of scene depth, we must initialize the depth distribution with a uniform distribution. However, we are able to trivially incorporate depth priors from external sources by initializing the depth distribution based on these priors, further improving the performance of adaptive gating.
We validated the performance of our algorithm by capturing data from real outdoor scenes. In order to capture data from scenes under direct sunlight, where photon pile-up is most severe, we moved our gated SPAD setup onto a trolley. Below are some images of our setup.
Hardware setup [top view]
Hardware setup [side view]
Captured Results
Comparison in direct sunlight
Our method provides better depth reconstruction accuracy while capturing data under direct sunlight at noon. Alternatively, our method facilitates faster acquisition (cutting total integration time by 67%) while maintaining equal or better depth reconstruction accuracy.
Comparison in bright indoors setting
Our method still performs better than previous state-of-the-art gating strategies even under conditions with less severe pile-up.
Comparison using external depth priors
The probabilistic basis of our method allows adaptive gating to be up-stream compatible. This means that we can utilize depth priors from external sources to initialize our adaptive gating algorithm. We experimented with using a off-the-shelf neural monocular depth estimator as prior to adaptive gating. Results show that using appropriate priors leads to better reconstruction accuracy and lower integration times.
For an in-depth description of the technology behind this work, please refer to our paper, supplementary material, CVPR poster, and CVPR talk video.
Ryan Po, Adithya Pediredla, and Ioannis Gkioulekas. "Adaptive Gating for Single-Photon 3D Imaging", CVPR 2022
Acknowledgments
We thank Akshat Dave, Ankit Raghuram, and Ashok Veeraraghavan for providing the high-power laser for experiments and invaluable assistance in its use; as well as Matthew O’Toole, David Lindell, and Dorian Chan for help with the SPAD sensor. This work was supported by NSF Expeditions award 1730147, NSF CAREER award 2047341, DARPA REVEAL contract HR0011-16-C-0025, and ONR DURIP award N00014-16-1-2906. Ioannis Gkioulekas is supported by a Sloan Research Fellowship.