MambaDepth: Enhancing Long-range Dependency for Self-Supervised Fine-Structured Monocular Depth Estimation

Abstract

In the field of self-supervised depth estimation, Convolutional Neural Networks (CNNs) and Transformers have traditionally been dominant. However, both architectures struggle with efficiently handling long-range dependencies due to their local focus or computational demands. To overcome this limitation, we present MambaDepth, a versatile network tailored for self-supervised depth estimation. Drawing inspiration from the strengths of the Mamba architecture, renowned for its adept handling of lengthy sequences and its ability to capture global context efficiently through a State Space Model (SSM), we introduce MambaDepth. This innovative architecture combines the U-Net’s effectiveness in self-supervised depth estimation with the advanced capabilities of Mamba. MambaDepth is structured around a purely Mamba-based encoder-decoder framework, incorporating skip connections to maintain spatial information at various levels of the network. This configuration promotes an extensive feature learning process, enabling the capture of fine details and broader contexts within depth maps. Furthermore, we have developed a novel integration technique within the Mamba blocks to facilitate uninterrupted connectivity and information flow between the encoder and decoder components, thereby improving depth accuracy. Comprehensive testing across the established KITTI dataset demonstrates MambaDepth’s superiority over leading CNN and Transformer-based models in self-supervised depth estimation task, allowing it to achieve state-of-the-art performance. Moreover, MambaDepth proves its superior generalization capacities on other datasets such as Make3D and Cityscapes. MambaDepth’s performance heralds a new era in effective long-range dependency modeling for self-supervised depth estimation. Code is available at https://github.com/ionut-grigore99/MambaDepth.

I Introduction

Accurate depth estimation from single image is an active research field to help computer reconstruct and understand the real scenes. It also has a large range of applications in diverse fields such as autonomous vehicles, robotics, augmented reality, etc. While supervised monocular depth estimation has been successful, it is suffering from the expensive access to ground truth. Furthermore, supervised depth estimators often face optimization challenges under sparse supervision and exhibit limited adaptability to new, unencountered scenarios.

Recently, self-supervised approaches have become increasingly prominent. Current strategies primarily focus on utilizing self-distillation techniques [45], incorporating depth hints [59], and employing multi-frame inference [60, 11]. Despite these advancements, a common shortfall is their inability to capture detailed scene intricacies, as illustrated in Figure 1. The challenge lies in effectively and efficiently learning these fine-grained structural details within a self-supervised framework.

CNNs [36] and Transformers [56] are two significant architectures in the realm of self-supervised depth estimation. CNNs, like Monodepth2 [17] and MiDaS [49], excel in hierarchical feature extraction with more efficiency in parameters compared to traditional fully connected networks. Their weight-sharing structure is key in identifying translational invariances and local patterns. Conversely, Transformers, initially developed for natural language processing, have adapted well to image processing tasks. Examples include Vision Transformer (ViT) [8] in image recognition and SwinTransformer [40] as a versatile vision task backbone. Unlike CNNs, Transformers process images not as spatial hierarchies but as sequences of patches, enhancing their global information capturing ability. This distinction has led to the emergence of hybrid architectures combining CNNs and Transformers, like Depthformer [38], TransDepth [61], and DPT [48].

Figure 1: Our method’s typical predictions on images from the KITTI dataset exhibit superior performance when compared to the classical Monodepth2 [17] and the contemporary attempts to use Transformers [71] or self-attention mechanism [72] in self-supervised monocular depth estimation. Notably, our approach excels in recovering intricate scene details.

Transformers, while adept at managing long-range dependencies, pose a significant computational load due to the self-attention mechanism’s quadratic scaling with input size. This is particularly challenging for high-resolution images as ones used for depth estimation. To address this, state space sequence models (SSMs) [21, 25], especially structured state space sequence models (S4) [24], have shown promise. They offer an efficient and effective approach for deep network construction, as seen in Mamba [22], which optimizes S4 with a selective mechanism and hardware-aware design. These models have shown potential in language, genomics, and also in vision tasks like image [43] and video classification [29]. Given that image patches and features can be treated as sequences [40, 8], this encourages the exploration of SSMs, particularly the Mamba blocks, to enhance U-Net long-range modeling capabilities.

Our main contributions are as follows:

• We propose MambaDepth, a versatile network designed for self-supervised depth estimation, utilizing a novel SSM-based structure which effectively captures both localized details and extensive dependencies within images and thus obtains fine-grained scene geometry of a single image. To the best of our knowledge, this is the first time SSMs are used for self-supervised depth estimation. It distinguishes itself from conventional Transformer-based models by offering linear feature size scaling, avoiding the Transformers’ usual quadratic complexity. Additionally, we have developed a novel integration technique for Mamba blocks that ensures seamless connectivity and information flow between the encoder and decoder components, thereby enhancing depth accuracy.

• Our extensive evaluations across the KITTI dataset show MambaDepth’s exceptional performance, significantly outperforming Transformer-based networks and existing self-supervised alternatives in accuracy and efficiency. Moreover, MambaDepth’s enhanced generalization is showcased through its application of a KITTI pre-trained model to diverse datasets, including successful zero-shot transfer to Make3D and Cityscapes.

• This breakthrough sets the stage for future network designs that efficiently and effectively handle long-range dependencies in self-supervised depth estimation.

II Related work

II-A Supervised Depth Estimation

Eigen and colleagues [10] were pioneers in adopting a learning-based strategy, employing a multiscale convolutional neural network combined with a scale-invariant loss function to estimate depth from a single image. This groundbreaking approach has since inspired a plethora of subsequent methodologies. Broadly, these techniques fall into two main categories: one views depth estimation as a problem of pixel-wise regression, as seen in works like [10], [28], [48], and [66]. The other approach treats it as a pixel-wise classification challenge, as demonstrated in studies [12] and [7]. While regression-based methods are capable of predicting continuous depth values, they often present optimization challenges. On the other hand, classification-based methods, though simpler to optimize, are limited to predicting discrete depth values. In an innovative attempt to harness the advantages of both regression and classification, certain studies, notably those referenced as [1, 31], have redefined depth estimation as a dual task involving both classification and regression at the pixel level. This method involves initially regressing a series of depth bins followed by a pixel-wise classification, where each pixel is assigned to its respective bin. The ultimate depth value is then derived as a linear amalgamation of the centers of these bins, with weights given by their respective probabilities. This hybrid technique has shown significant enhancements in terms of accuracy.

Figure 2: Overview of our self-supervised framework. Our proposed MambaDepth adopts a U-Net architecture, leveraging MambaDepth blocks from encoder to obtain low-resolution feature maps of the current frame $I_{t}$. Subsequently, low-resolution feature maps traverse successive MambaDepth blocks from the decoder together with skip connections in order to obtain disparities after applying a final Sigmoid layer. The predicted disparities are then upsampled at various scales to match the original input resolutions. Additionally, a standard pose network utilizes temporally adjacent frames $I_{t}$ and $I_{t-1}$ as input, yielding relative pose $T_{t-1\rightarrow t}$ as output. The camera pose is solely required during training for conducting differentiable warping. In line with numerous prior studies, we employ pixels from frame $I_{t-1}$ to reconstruct frame $I_{t}$ using the depth map $D_{t}$ and relative pose $T_{t-1\rightarrow t}$ through a differentiable warping process [30]. The loss function is formulated based on the differences between the warped image $I_{t-1\rightarrow t}$ and the source image $I_{t}$.

II-B Self-supervised Depth Estimation

In scenarios lacking direct ground truth data, self-supervised models are often developed by leveraging either the temporal consistency found in monocular video sequences, as explored in studies [17, 67], or the left-right consistency observed in stereo image pairs, a concept investigated in references [46, 14, 16].

Monocular Training Approach: This method derives supervision from the congruence between a synthesized scene view from a reference frame and the actual view from a source frame. A notable example, the SfMLearner [46], synchronizes the training of a DepthNet and a separate PoseNet using a photometric loss function. Building on this foundational approach, numerous enhancements have been proposed. These include robust image-level reconstruction losses [53, 20], feature-level reconstruction losses [53, 64], incorporation of auxiliary information during training [34, 59], strategies to address the dynamic objects that disrupt static scene assumptions [17, 20, 50, 57, 5, 2, 3, 33, 37, 63], and additional constraints [62, 63, 50, 4, 18, 26, 69].

Stereo Training Method: Here, synchronized stereo image pairs are used, focusing on predicting a disparity map [52], which is effectively the inverse of a depth map. With known relative camera poses, the model’s task is simplified to disparity map prediction. Garg et al. [14] pioneered this with a self-supervised monocular depth estimator, applying a photometric consistency loss between stereo pairs. Subsequent improvements include implementing left-right consistency [16] and temporal consistency in videos [64]. Garg et al. [13] further refined this by enabling the prediction of continuous disparity values. Stereo-based methods have evolved to include semi-supervised data [35, 41], auxiliary information usage [59], exponential probability volumes [19], and self-distillation techniques [47, 27, 45]. Stereo views offer an ideal reference for supervision and can also be instrumental in deriving absolute depth scales.

However, existing self-supervised methods still struggle with producing high-fidelity depth maps. Current techniques primarily rely on immediate visual features or utilize Transformer [8] enhanced high-level visual representations, often neglecting the critical role of pixel-level geometric cues that could significantly enhance model performance and generalization abilities.

II-C State Space Models

State Space Sequence Models (SSMs) [21] represent a category of systems that transform a one-dimensional function or sequence $u(t)$ into $y(t)$. They are described by the following linear Ordinary Differential Equation (ODE):

$$x^{\prime}(t)=Ax(t)+Bu(t),$$ (1)

$$y(t)=Cx(t)+Du(t).$$ (2)

In this equation, $A\in\mathbb{R}^{N\times N}$ is the state matrix, and $B,C\in\mathbb{R}^{N}$ are its parameters and skip connection $D\in\mathbb{R}^{1}$, with $x(t)\in\mathbb{R}^{N}$symbolizing the implicit latent state. SSMs possess several advantageous properties, such as linear complexity per time step and the ability for parallel computations, which aid in efficient training. Yet, standard SSMs generally require more memory compared to equivalent CNNs and face issues like vanishing gradients during training, limiting their widespread use in sequence modeling.

The evolution of SSMs led to the creation of Structured State Space Sequence Models (S4) [24], which notably enhance the basic SSM framework. S4 achieves this by applying structured designs to the state matrix $A$ and incorporating an efficient algorithm. The state matrix in S4 is specifically developed and initialized using the High-Order Polynomial Projection Operator (HIPPO) [23], facilitating the construction of deep sequence models that are both rich in capability and adept at long-range reasoning. Remarkably, S4 has outperformed Transformers [56] in the demanding Long Range Arena Benchmark [54].

Mamba [22] represents a further advancement in SSMs, especially in discrete data modeling, such as text and genomic sequences. Mamba introduces two significant enhancements. Firstly, it incorporates an input-specific selection mechanism, differing from traditional, invariant SSMs. This mechanism filters information efficiently by customizing SSM parameters based on input data. Secondly, Mamba employs a hardware-optimized algorithm, which scales linearly with sequence length and uses a scanning process for recurrent computation, enhancing speed on contemporary hardware. Mamba’s architecture, which combines SSM blocks with linear layers, is notably more streamlined. It has achieved top-tier results in various long-sequence fields, including language and genomics, demonstrating considerable computational efficiency in both training and inference phases.

III Method

Figure 3: Overview of MambaDepth architecture. The MambaDepth structure includes an encoder, bottleneck, a decoder, and skip connections. Each of these components – the encoder, bottleneck, and decoder – is built using the MD block.

III-A Self-supervised framework

In this section we describe the framework of our model and describe how we provide the supervisory signal during the training of our model. Fundamentally, our method is a form of Structure from Motion (SfM), where the monocular camera is moving within a rigid environment to provide multiple views of that scene. Our framework is built upon Monodepth2 [17].

Let $I_{t}\in\mathbb{R}^{H\times W\times 3},t\in\left\{-1,0,1\right\}$ be a frame in a monocular video sequence captured by a moving camera, where $t$ is the frame time index. Similarly, let $D_{t}\in\mathbb{R}^{H\times W}$ denote the depth map corresponding to image $I_{t}$. The camera pose changes from time $0$ to time $t,t\in\left\{-1,1\right\}$ is encoded by the $3\times 3$ rotation matrix $R_{t}$ and the translation vector $t_{t}$. We obtain the $4\times 4$ camera transformation matrix thus:

$$M_{t}=\begin{bmatrix}R_{t}&t_{t}\\ 0&1\end{bmatrix}.$$ (3)

Our aim is to train two CNN networks to simultaneously estimate the pose of the camera, and the structure of the scene, respectively:

$$M=\theta_{pose}(I_{t}),$$ (4)

$$D=\theta_{depth}(I_{t}).$$ (5)

Self-supervised depth prediction reformulates the learning task as a novel view-synthesis problem. Specifically, during training, we let the coupled network synthesize the photo-consistency appearance of a target frame from another viewpoint of the source frame. We treat the depth map as an intermediate variable to constrain the network to complete the image synthesis task.

Let $(u,v)\in\mathbb{R}^{2}$ be the calibrated coordinates of a pixel in image $I_{0}$. In this case, let the origin $(0,0)$ be the top-left of the image. In the process of imaging, a 3D point $(X,Y,Z)\in\mathbb{R}^{3}$ projects onto $(u,v)$ through a perspective projection operator.

Suppose that the transformation matrix $M_{t}$ encodes the pose change of the camera from time $0$ to time $t$ and (6) is the perspective projection operator:

$$\pi(X,Y,Z)=\left(f_{x}\frac{X}{Z}+c_{x},f_{y}\frac{Y}{Z}+c_{y}\right)=(u,v),$$ (6)

where $(f_{x},f_{y},c_{x},c_{y})$ are the camera intrinsic parameters. Therefore, given a depth map $D(u,v)$, a 2D image point $(u,v)$ backprojects to a 3D point $(X,Y,Z)$ through backprojection operator:

$$\pi^{-1}(u,v,D(u,v))=D(u,v)\left(\frac{u-c_{x}}{f_{x}},\frac{v-c_{y}}{f_{y}}\right)=(X,Y,Z).$$ (7)

Then the corresponding pixels in image $I_{t}$ can be computed as:

$$(u^{\prime},v^{\prime})=\pi(M_{t}\pi^{-1}(u,v,D(u,v)))=g(u,v|D(u,v),M_{t}).$$ (8)

We project the pixels of an image to form a novel synthetic view (8). However, the projected coordinates $(u^{\prime},v^{\prime})$ are continuous values. To obtain $I_{s}(u,v)$ we include a differentiable bilinear sampling mechanism, as proposed in spatial transformer networks [30]. We can now linearly interpolate the values of the 4-pixel neighbors (top-left, top-right, bottom-left, bottom-right) of $I(u^{\prime},v^{\prime})$ to give the RGB intensities as follows:

$$I^{s}(u,v)=\sum_{u}\sum_{v}w^{uv}I(u^{\prime},v^{\prime}),$$ (9)

where $w^{uv}$ is linearly proportional to the spatial proximity between $(u,v)$ and $(u^{\prime},v^{\prime})$, and $\sum_{u,v}w^{uv}=1$.

Figure 4: The scan expanding and scan merging operations in SS2D. In the SS2D method, input patches follow four distinct scanning paths. Each sequence is then independently processed by separate S6 blocks. Finally, the results are combined to create a 2D feature map, which serves as the final output.

III-B MambaDepth

In this section, we elaborate the design details of the core components in the MambaDepth.

The Mamba framework has shown remarkable efficacy in processing various kinds of discrete data. However, its application in image data processing, especially in the field of self-supervised depth estimation, has not been fully explored. [42] proposes U-Mamba, a hybrid CNN-SSM architecture, to handle the long-range dependencies in biomedical image segmentation and [68] build a pure SSM-based model, which can be adopted as a generic vision backbone, but their efficiency are not yet fully understood at a large scale. Images, fundamentally discrete samples from continuous signals, can be transformed into extended sequences. This characteristic suggests the potential of using Mamba’s linear scaling benefits to improve the capability of UNet architecture in modeling extensive range dependencies. While image processing using Transformers, like ViT and SwinTransformer, has seen success, their application is limited by the significant computational demands for large images due to the self-attention mechanism’s quadratic complexity. This challenge presents an opportunity to utilize Mamba’s linear scaling to bolster UNet capacity for long-range dependency modeling.

In self-supervised depth estimation, Monodepth2 [17] and its derivatives, recognized for their symmetric encoder-decoder structure, are predominant. This structure is adept at extracting multi-level image features through convolutional methods. However, the design is constrained in its ability to capture long-range dependencies in images, as the convolutional kernels focus on local areas. Each convolutional layer only processes features within its limited receptive field. Although skip connections aid in merging detailed and abstract features, they primarily enhance local feature combination, not the modeling of extensive range dependencies.

MambaDepth, a novel design, integrates the strengths of Monodepth2 [17] and Mamba [22] to comprehensively understand global contexts in self-supervised depth estimation. Figure 3 showcases the structure of MambaDepth, which is distinct in its composition, featuring an embedding layer, encoder, decoder, disparity heads, and straightforward skip connections, marking a departure from the classical designs often found in prior work.

In the initial stage, the embedding layer segments the input image, denoted as $x$ with dimensions $\mathbb{R}^{H\times W\times 3}$, into discrete $4\times 4$ patches. These patches are then transformed to a predefined dimension $C$ (typically set to 96), resulting in a reshaped image representation, $x^{\prime}$, with dimensions $\mathbb{R}^{\frac{H}{4}\times\frac{W}{4}\times C}$. $x^{\prime}$ is normalized using Layer Normalization before its progression into the encoder for the extraction of features. The encoder itself is structured into four phases, incorporating our newly introduced feature fusion step at the conclusion of the first three phases to compact the spatial dimensions and amplify the channel depth, employing a configuration of $[2,2,2,2]$ MD blocks and channel dimensions scaling from $[C,2C,4C,8C]$ through each phase.

Conversely, the decoder reverses this process over its four stages, using our introduced feature decomposition technique at the start of the final three stages to enlarge spatial dimensions and condense the channel count. Here, the arrangement of MD blocks is $[2,2,2,2]$, with channel dimensions inversely scaling from $[8C,4C,2C,C]$. The decoder culminates in 4 disparity heads that upscale the feature dimensions by a factor of four through feature decomposition, followed by a projection layer that adjusts the channel count to align with the target of self-supervised depth estimation, which is processed through a convolutional layer and a Sigmoid layer to generate the final depth map.

Skip connections within this architecture are implemented via a simple addition operation, purposely designed to avoid the incorporation of extra parameters, thus maintaining the model’s efficiency and simplicity.

III-C MD block

Figure 5: The detailed structure of the MD (MambaDepth) Block.

Method	Train	Params	H$\times$W	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSESlog $\downarrow$	$\delta<1.25$$\uparrow$	$\delta<1.25^{2}$$\uparrow$	$\delta<1.25^{3}$$\uparrow$
MonoDepth2 [17]	M	34M	192$\times$640	0.110	0.831	4.642	0.187	0.883	0.962	0.982
CaDepth-Net [72]	M	58M	192$\times$640	0.105	0.769	4.535	0.181	0.892	0.964	0.983
PackNet-SfM [76]	M	128M	192$\times$640	0.107	0.785	4.612	0.185	0.887	0.962	0.982
HR-Depth [70]	M	14M	192$\times$640	0.109	0.792	4.632	0.185	0.884	0.962	0.983
Lite-Mono [71]	M	8M	192$\times$640	0.107	0.765	4.561	0.183	0.886	0.963	0.983
DynamicDepth [11]	M	-	192$\times$640	0.096	0.720	4.458	0.175	0.897	0.964	0.984
MonoViT [65]	M	27M	192$\times$640	0.099	0.708	4.372	0.175	0.900	0.967	0.984
MambaDepth (Ours)	M	30M	192$\times$640	0.097	0.706	4.370	0.172	0.907	0.970	0.986
MonoDepth2 [17]	MS	34M	320$\times$1024	0.106	0.806	4.630	0.193	0.876	0.958	0.980
MonoDepth2 [17]	M	34M	320$\times$1024	0.115	0.882	4.701	0.190	0.879	0.961	0.982
HR-Depth [70]	MS	14M	320$\times$1024	0.101	0.716	4.395	0.179	0.899	0.966	0.983
HR-Depth [70]	M	14M	320$\times$1024	0.106	0.755	4.472	0.181	0.892	0.966	0.984
EPCDepth [45]	S	-	320$\times$1024	0.091	0.646	4.207	0.176	0.901	0.966	0.983
Depth Hints [59]	MS	-	320$\times$1024	0.098	0.702	4.398	0.183	0.887	0.963	0.983
Depth Hints [59]	S	-	320$\times$1024	0.096	0.710	4.393	0.185	0.890	0.962	0.981
CADepth-Net [72]	MS	58M	320$\times$1024	0.096	0.964	4.264	0.173	0.908	0.968	0.984
CADepth-Net [72]	M	58M	320$\times$1024	0.102	0.734	4.407	0.178	0.898	0.966	0.984
MT-SfMLearner [77]	M	-	320$\times$1024	0.104	0.799	4.547	0.181	0.893	0.963	0.982
Lite-Mono [71]	M	8M	320$\times$1024	0.097	0.710	4.309	0.174	0.905	0.967	0.984
MonoViT [65]	M	27M	320$\times$1024	0.096	0.714	4.292	0.172	0.908	0.968	0.984
MambaDepth (Ours)	M	30M	320$\times$1024	0.095	0.634	3.402	0.169	0.914	0.968	0.985

Table I: Performance comparison on KITTI [15] eigen benchmark. Best results are marked in bold.

Method	Train	Params	H$\times$W	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSESlog $\downarrow$	$\delta<1.25$$\uparrow$	$\delta<1.25^{2}$$\uparrow$	$\delta<1.25^{3}$$\uparrow$
MonoDepth2 [17]	M	34M	192$\times$640	0.090	0.545	3.942	0.137	0.914	0.983	0.995
HR-Depth [70]	M	14M	192$\times$640	0.085	0.471	3.769	0.130	0.919	0.985	0.996
PackNet-SfM [76]	M	128M	192$\times$640	0.078	0.420	3.485	0.121	0.931	0.986	0.996
CADepth-Net [72]	M	58M	192$\times$640	0.080	0.450	3.649	0.124	0.927	0.986	0.996
DIFFNet [75]	M	-	192$\times$640	0.076	0.412	3.494	0.119	0.935	0.988	0.996
MonoViT [65]	M	27M	192$\times$640	0.075	0.389	3.419	0.115	0.938	0.989	0.997
MambaDepth (Ours)	M	30M	192$\times$640	0.073	0.386	3.415	0.112	0.941	0.991	0.997
MonoDepth2 [17]	MS	34M	320$\times$1024	0.091	0.531	3.742	0.135	0.916	0.984	0.995
MonoDepth2 [17]	S	34M	320$\times$1024	0.077	4.455	3.489	0.119	0.933	0.988	0.996
DepthHints [59]	S	-	320$\times$1024	0.074	0.364	3.202	0.114	0.936	0.989	0.997
MambaDepth (Ours)	M	30M	320$\times$1024	0.072	0.410	3.490	0.114	0.940	0.989	0.996

Table II: Performance comparison using KITTI improved ground truth [55]. Best results are marked in bold.

At the heart of MambaDepth lies the MD (Mamba Depth) module, which is an adaptation from VMamaba [39], as illustrated in Figure 5. This module begins with Layer Normalization of the input, which then bifurcates into two distinct pathways. The initial pathway channels the input through a linear transformation and an activation phase, whereas the second pathway subjects the input to a sequence involving a linear transformation, depthwise separable convolution, and activation, before directing it to the 2D-Selective-Scan (SS2D) component for advanced feature extraction. Following this, the extracted features undergo Layer Normalization, are then merged with the output of the first pathway through element-wise multiplication, and finally are integrated using a linear transformation. This process is augmented with a residual connection, culminating in the output of the MD block. For activation, the SiLU function is consistently utilized throughout this study.

The SS2D mechanism comprises three essential stages: an operation to expand the scan, an S6 block for processing, and a merging operation for the scans. As shown in Figure 4, the expansion phase unfolds the input image in four orientations (diagonally and anti-diagonally) into sequences. These sequences are then refined by the S6 block, a procedure that meticulously scans information from all directions to extract a comprehensive range of features. Following this, the sequences are recombined through a merging operation, ensuring the output image is resized back to the original dimensions. The innovative S6 block, evolving from Mamba and building upon the S4 structure, introduces a selective filter that dynamically adjusts to the input by fine-tuning the parameters of the State Space Model (SSM). This adjustment allows the system to selectively focus on and preserve relevant information, while discarding what is unnecessary.

Figure 6: Qualitative results on the KITTI eigen benchmark.

III-D Loss function

Objective functions. In line with the methodologies described in [16, 17], we adopt the conventional photometric loss $pe$, which is a combination of $L1$ and $SSIM$ losses:

$$pe(I_{a},I_{b})=\frac{\alpha}{2}(1-SSIM(I_{a},I_{b}))+(1-\alpha)\left\|I_{a}-I_{b}\right\|_{1}.$$ (10)

To ensure proper depth regularization in areas lacking texture, we employ an edge-aware smooth loss, applied in the following manner:

$$L_{s}=\left|\partial_{x}d_{t}^{*}\right|e^{-\left|\partial_{x}I_{t}\right|}+\left|\partial_{y}d_{t}^{*}\right|e^{-\left|\partial_{y}I_{t}\right|}.$$ (11)

Method	Train	Params	H$\times$W	AbsRel$\downarrow$	SqRel$\downarrow$	RMSE$\downarrow$	RMSESlog$\downarrow$	$\delta<1.25$$\uparrow$	$\delta<1.25^{2}$$\uparrow$	$\delta<1.25^{3}$$\uparrow$
Struct2Depth 2 [74]	M	-	416$\times$128	0.145	1.737	7.280	0.205	0.813	0.942	0.976
MonoDepth2 [17]	MS	34M	416$\times$128	0.129	1.569	6.876	0.187	0.849	0.957	0.983
ManyDepth [60]	MS	37M	416$\times$128	0.114	1.193	6.223	0.170	0.875	0.967	0.989
MambaDepth (Ours)	M	30M	416$\times$128	0.112	1.186	6.226	0.167	0.876	0.968	0.990

Table III: Cityscapes results. Best results are marked in bold.

Masking Strategy. In real-world settings, scenarios featuring stationary cameras and moving objects can disrupt the usual assumptions of a moving camera and static environment, negatively impacting the performance of self-supervised depth estimators. Previous studies have attempted to enhance depth prediction accuracy by incorporating a motion mask, which addresses moving objects using scene-specific instance segmentation models. However, this approach limits their applicability to new, unencountered scenarios. To maintain scalability, our method eschews the use of a motion mask for handling moving objects. Instead, we adopt the auto-masking strategy outlined in [17], which filters out static pixels and areas of low texture that appear unchanged between two consecutive frames in a sequence. This binary mask $\mu$ is calculated as per (12), employing the Iverson bracket notation:

$$\mu=[\underset{t^{\prime}}{\mathrm{min}}\,pe(I_{t},I_{t^{\prime}->t})<\underset{t^{\prime}}{\mathrm{min}}\,pe(I_{t},I_{t^{\prime}})].$$ (12)

Final Training Loss. We formulate the final loss by combining our per-pixel smooth loss with the masked photometric losses:

$$L=\mu L_{p}+\lambda L_{s}.$$ (13)

IV Experiments

The effectiveness of MambaDepth is assessed using the public dataset KITTI. We measure the model’s performance using several established metrics from [9].

IV-A Datasets and Experimental Protocol

KITTI [15] dataset, known for its stereo image sequences, is widely utilized in self-supervised monocular depth estimation. We employ the Eigen split [9], comprising about 26,000 images for training and 697 for testing. Our approach with MambaDepth involves training it from the beginning on KITTI under minimal conditions: it operates solely with auto-masking [17], without additional stereo pairs or auxiliary data. For testing purposes, we maintain a challenging scenario by using only a single frame as input, in contrast to other methods that might use multiple frames to enhance accuracy.

Cityscapes [6] dataset, noted for its complexity and abundance of moving objects, serves as a testing ground to assess the adaptability of MambaDepth. To this end, we conduct a zero-shot evaluation on Cityscapes, utilizing a model pre-trained on KITTI. It is crucial to highlight that, unlike many competing approaches, we do not employ a motion mask in our evaluation. For data preparation, we follow the same preprocessing procedures outlined in [67], which are also adopted by other baselines, converting the image sequences into triplets.

Make3D [51]. To assess the capability of MambaDepth to generalize to new, previously unseen images, the model, initially trained on the KITTI dataset, was subjected to a zero-shot evaluation using the Make3D dataset. Furthermore, supplementary visualizations of depth maps are provided.

IV-B Implementation Details

We developed our model using the PyTorch framework [44]. It was trained on eight NVIDIA Tesla V100-SXM2 GPUs, with a batch size of $8$. We pre-trained the architecture on the ImageNet-1k dataset, subsequently using these pre-trained weights to initialize both the encoder and decoder components of the model. In line with the approach in [17], we applied color and flip augmentations to the images during training. Both DepthNet and PoseNet were trained using the Adam Optimizer [32], with $\beta_{1}$ set at 0.9 and $\beta_{2}$ at 0.999, DepthNet being also trained concurrently. The learning rate starts at $1e-4$ and reduces to $1e-5$ after $15$ epochs. We configured the SSIM weight at $\alpha=0.85$ and the weight for the smooth loss term at $\lambda=1e-3$.

Figure 7: Model parameters and performance with respect to the absolute relative difference (Abs Rel) on the KITTI Eigen test set. Our model is more efficient and accurate.

IV-C KITTI Results

We evaluate our MambaDepth using the standard KITTI Eigen split [9], which consists of 697 images paired with raw LiDAR scans. Improved ground truth labels [55] are available for 652 of these images. To address monocular scale ambiguity in depth models trained on video sequences, we scale the estimated depth by the per-image median ground truth [67].

Table I presents the results achieved by state-of-the-art self-supervised frameworks, processing either low-resolution (640$\times$192) or high-resolution (1024$\times$320) images. MambaDepth significantly outperforms existing state-of-the-art methods across all training resolutions on all metrics, some of them trained solely on stereo videos ([59, 45]) or with binocular videos while MambaDepth being trained entirely on monocular videos. Notably, MambaDepth substantially surpasses MonoViT [65], Lite-Mono [71], and MT-SfMLearner [77], the best contemporary attempts to use Transformers or self-attention mechanism in self-supervised monocular depth estimation.

Table II shows the same metrics computed using the improved ground truth labels for images processed at 640$\times$192 resolution and 1024$\times$320 resolution. Again, MambaDepth consistently demonstrates higher accuracy.

Figure 6 compares MambaDepth with some of its competitors, illustrating that our model achieves significantly lower RMSE. This comparison highlights MambaDepth’s superior capability in modeling long-range relationships between objects compared to existing models.

Also the model is more efficient and accurate in terms of computations costs, as depicted in Figure 8, where we compare the absolute relative difference against Giga Multiply-Add Calculations per Second on the KITTI Eigen test set for more existing state-of-the-art methods. As for parameters comparison, details are present in Figure 7.

Figure 8: Comparisons about computational costs. We compare AbsRel against Giga Multiply-Add Caculation per Second (GMACs) on the KITTI Eigen test set. Our model is more efficient and accurate.

IV-D Cityscapes Results

To evaluate the generalization of MambaDepth, we conducted zero-shot evaluation. For this, we utilized the model pretrained on KITTI for images processed at 416$\times$128 resolution. The results, summarized in Table III, indicate that MambaDepth performs exceptionally well being entirely trained with monocular videos, unlike most baselines in Table III trained with a combination of monocular videos and stereo pairs. Notably, MambaDepth achieves a 1.75% error reduction compared to the well-known ManyDepth [60], which utilizes two frames (the previous and current) as input. These findings underscore the superior generalization ability of MambaDepth.

IV-E Make3D Results

To further assess the generalization capability of MambaDepth, we conducted a zero-shot evaluation on the Make3D dataset [51] using the pretrained weights from KITTI. Adhering to the evaluation protocol described in [16], we tested on a center crop with a 2:1 aspect ratio. As illustrated in Table IV and Figure 9, MambaDepth outperformed the baselines, delivering sharp depth maps with more precise scene details. These results highlight the exceptional zero-shot generalization ability of our model.

Method	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSESlog $\downarrow$
Zhou [67]	0.383	5.321	10.470	0.478
DDVO [73]	0.387	4.720	8.090	0.204
MonoDepth2 [17]	0.322	3.589	7.417	0.163
CADepth-Net [72]	0.312	3.086	7.066	0.159
MambaDepth (Ours)	0.307	2.405	6.858	0.153

Table IV: Make3D results. Best results are marked in bold.

Figure 9: Qualitative Make3D results (Zero-shot). Predictions by CA-Depth [72], MonoDepth2 [17] and our MambaDepth.

IV-F Ablation study

Here, we explore the impact of initialization for MambaDepth using KITTI dataset. We initialize MambaDepth with and without weights pretrained on ImageNet. The results, displayed in Table V, suggest that stronger pretrained weights markedly improve MambaDepth’s subsequent effectiveness, highlighting the significant role these initial weights play.

Weights initialization	AbsRel $\downarrow$	SqRel $\downarrow$	RMSE $\downarrow$	RMSESlog $\downarrow$
Xavier	0.114	0.881	4.700	0.191
ImageNet pretraining	0.095	0.634	3.402	0.169

Table V: Ablation study on weights initialization for MambaDepth on KITTI eigen benchmark. Best results are marked in bold.

V Conclusions

In our study, we re-examine the challenges of self-supervised monocular depth estimation and present a novel and efficient approach, which we have named MambaDepth. This method is designed to overcome the challenges in capturing long-range dependencies, a limitation observed in the localized nature of CNNs and the computational intensity of Transformers. MambaDepth is a purely Mamba block-based U-Net-style network for self-supervised monocular depth estimation. It achieves outstanding state-of-the-art performance on the KITTI dataset. Additionally, we showcase the enhanced generalizability of our model in various settings. Our findings position MambaDepth as a leading contender for future advanced self-supervised depth estimation networks.