M${^2}$Depth: Self-supervised Two-Frame Multi-camera Metric Depth Estimation

Overview

• This paper presents a novel depth estimation network called M²Depth that uses two-frame images from multiple cameras to predict reliable scale-aware surrounding depth in autonomous driving scenarios.
• Unlike previous methods, M²Depth combines spatial and temporal information to produce high-quality depth maps, and it also leverages neural priors to improve depth edge detection.
• Experiments on the nuScenes and DDAD datasets show that M²Depth achieves state-of-the-art performance in depth estimation for autonomous driving.

Plain English Explanation

M²Depth is a new depth estimation system that uses images from multiple cameras taken at slightly different time points to create accurate 3D depth information around a self-driving car. Previous methods either used images from a single camera at one time point or images from multiple cameras at the same time, but M²Depth combines both spatial and temporal information to get the best of both worlds.

The key innovation is the way M²Depth processes the input images. First, it creates cost volumes that capture the spatial and temporal relationships between the images. Then, it fuses these spatial and temporal cost volumes together to get a robust 3D representation. Finally, it combines this with neural priors, which are general patterns the model has learned, to refine the depth edges and distinguish the foreground from the background.

This produces depth maps that are highly accurate and can give the self-driving car a clear understanding of its 3D surroundings, which is crucial for safe navigation. The researchers showed that M²Depth outperforms previous state-of-the-art methods on standard autonomous driving benchmarks.

Technical Explanation

M²Depth is designed to leverage two-frame images from multiple cameras to estimate accurate, scale-aware depth in autonomous driving scenarios. Unlike prior work that used either single-view images at one time point or multi-view images at the same time point, M²Depth combines spatial and temporal information to produce high-quality depth maps.

The core of the M²Depth architecture is a spatial-temporal fusion module that integrates cost volumes constructed in the spatial and temporal domains. The spatial cost volume captures the geometric relationships between the multi-view images, while the temporal cost volume encodes the motion dynamics between the two time frames. Fusing these two cost volumes enables M²Depth to effectively leverage both spatial and temporal cues.

Additionally, M²Depth combines the neural prior from SAM features with its internal features to reduce ambiguity between the foreground and background, and to strengthen the depth edges. This helps M²Depth produce sharper, more reliable depth predictions.

Extensive experiments on the nuScenes and DDAD benchmarks demonstrate that M²Depth achieves state-of-the-art performance in depth estimation for autonomous driving, surpassing previous monocular and multi-view depth estimation methods.

Critical Analysis

The paper provides a thorough evaluation of M²Depth's performance, comparing it to a wide range of prior work on standard autonomous driving depth estimation benchmarks. The results clearly show the benefits of the spatial-temporal fusion approach and the incorporation of neural priors.

However, the paper does not deeply discuss the potential limitations or failure cases of the M²Depth system. For example, it is unclear how well the method would perform in challenging lighting conditions, such as very low light or strong shadows, or in complex urban environments with many occlusions. Additionally, the reliance on multiple camera inputs may limit the applicability of M²Depth to situations where only a single camera is available.

Further research could explore ways to make M²Depth more robust to diverse environmental conditions, or to adapt the approach to work with monocular or sparse multi-view inputs. Investigating the computational and memory requirements of the system would also be valuable, as efficiency is a key concern for real-world autonomous driving applications.

Conclusion

This paper presents a novel depth estimation network called M²Depth that leverages two-frame multi-camera inputs to predict reliable, scale-aware depth in autonomous driving scenarios. By fusing spatial and temporal cost volumes, and combining neural priors with internal features, M²Depth is able to outperform previous state-of-the-art depth estimation methods on standard benchmarks.

The ability to accurately estimate 3D depth from camera inputs is a critical capability for self-driving cars, enabling them to understand and navigate their surroundings safely. The innovations in M²Depth represent an important step forward in this area, and the techniques could potentially be applied to other computer vision tasks that require robust 3D perception.