Repurposing Diffusion-Based Image Generators for Monocular Depth Estimation

Abstract

Monocular depth estimation is a fundamental computer vision task. Recovering 3D depth from a single image is geometrically ill-posed and requires scene understanding, so it is not surprising that the rise of deep learning has led to a breakthrough. The impressive progress of monocular depth estimators has mirrored the growth in model capacity, from relatively modest CNNs to large Transformer architectures. Still, monocular depth estimators tend to struggle when presented with images with unfamiliar content and layout, since their knowledge of the visual world is restricted by the data seen during training, and challenged by zero-shot generalization to new domains. This motivates us to explore whether the extensive priors captured in recent generative diffusion models can enable better, more generalizable depth estimation. We introduce Marigold, a method for affine-invariant monocular depth estimation that is derived from Stable Diffusion and retains its rich prior knowledge. The estimator can be fine-tuned in a couple of days on a single GPU using only synthetic training data. It delivers state-of-the-art performance across a wide range of datasets, including over 20% performance gains in specific cases. Project page: https://marigoldmonodepth.github.io.

1 Introduction

Monocular depth estimation aims to transform a photographic image into a depth map, i.e., regress a range value for every pixel. The task arises whenever the 3D scene structure is needed, and no direct range or stereo measurements are available. Clearly, undoing the projection from the 3D world to a 2D image is a geometrically ill-posed problem and can only be solved with the help of prior knowledge, such as typical object shapes and sizes, likely scene layouts, occlusion patterns, etc. In other words, monocular depth implicitly requires scene understanding, and it is no coincidence that the advent of deep learning brought about a leap in performance. Following the seminal work of Eigen et al. [12], depth estimation is nowadays cast as neural image-to-image translation and learned in a supervised fashion using collections of paired and well-aligned RGB and depth images. Early methods of this type were limited to a narrow domain defined by their training data, often indoor [45] or driving [16] scenes. More recently, there has been a quest to train generic depth estimators that can be either used off-the-shelf across a broad range of scenes or fine-tuned to a specific application scenario with a small amount of data. These models generally follow the strategy first employed by MiDAS [33] to achieve generality, namely to train a high-capacity model with data sampled from many different RGB-D datasets (respectively, domains). The latest developments include moving from convolutional encoder-decoder networks [33] to increasingly large and powerful vision transformers [34], and training on more and more data and with additional surrogate tasks [11] to amass even more knowledge about the visual world, and consequently to produce better depth maps. Importantly, visual cues for depth depend not only on the scene content but also on the (generally unknown) camera intrinsics [56]. For general in-the-wild depth estimation, it is often preferred to estimate affine-invariant depth (i.e., depth values up to a global offset and scale), which can also be determined without objects with known sizes that could serve as “scale bars”.

The intuition behind our work is the following: Modern image diffusion models have been trained on internet-scale image collections specifically to generate high-quality images across a wide array of domains [36, 3, 39]. If the cornerstone of monocular depth estimation is indeed a comprehensive, encyclopedic representation of the visual world, then it should be possible to derive a broadly applicable depth estimator from a pretrained diffusion model. In this paper, we set out to explore this option and develop Marigold, a latent diffusion model (LDM) based on Stable Diffusion [36], along with a fine-tuning protocol to adapt the model for depth estimation. The key to unlocking the potential of a pretrained diffusion model is to keep its latent space intact. We find this can be done efficiently by modifying and fine-tuning only the denoising U-Net. Turning Stable Diffusion into Marigold requires only synthetic RGB-D data (in our case, the Hypersim [35] and Virtual KITTI [6] datasets) and a few GPU days on a single consumer graphics card. Empowered by the underlying diffusion prior of natural images, Marigold exhibits excellent zero-shot generalization: Without ever having seen real depth maps, it attains state-of-the-art performance on several real datasets. To summarize, our contributions are:

1. A simple and resource-efficient fine-tuning protocol to convert a pretrained LDM image generator into an image-conditional depth estimator;

2. Marigold, a state-of-the-art, versatile monocular depth estimation module that offers excellent performance across a wide variety of natural images.

2 Related Work

2.1 Monocular Depth

At the technical level, monocular depth estimation is a dense, structured regression task. The pioneering work of Eigen et al. [12] introduced a multi-scale network and showed that the result can be converted to metric depth for a dataset recorded with a single sensor. Successive improvements have come from various fronts, including ordinal regression [13], planar guidance maps [22], neural conditional random fields [57], vision transformers [52, 25, 1], a piecewise planarity prior [32], first-order variational constraints [26] and variational autoencoders [30]. Some authors treat depth estimation as a combined regression-classification task, using various binning strategies like AdaBins [4] or BinsFormer [24] to discretize depth range. A notable recent trend involves training generative models, especially diffusion models [47, 18] for monocular depth estimation [20, 10, 42, 41]. Recently, a few works like Metric3D [56] and ZeroDepth [17] revisited the depth estimation by explicitly feeding camera intrinsics as additional input.

Estimating depth “in the wild” refers to methods that are successful across a wide range of (possibly unfamiliar) settings, a particularly challenging task. It has been addressed by constructing large and diverse depth datasets and designing algorithms to handle that diversity. Depth-in-the-wild (DIW) [7] was perhaps the earliest work to introduce an uncontrolled dataset and to predict relative (ordinal) depth for it. OASIS [8] introduced relative depth and normals to better generalize across scenes. However, relative depth predictions (depth ordering) are of limited use for many downstream tasks, which has led several authors to consider affine-invariant depth. In that setting, depth is estimated up to an unknown (global) offset and scale. It offers a viable compromise between the ordinal and metric cases: on the one hand, it can handle general scenes consisting of unfamiliar objects; on the other hand, depth differences between different objects or scene parts are still geometrically meaningful relative to each other. MegaDepth [23] and DiverseDepth [54] utilize large internet photo collections to train models that can adapt to unseen data, while MiDaS [33] achieves generality by training on a mixture of multiple datasets. The step from CNNs to vision transformers has further boosted performance, as evidenced by DPT (MiDaS v3) [34] and Omnidata [11]. LeReS [55] proposed a two-stage framework that first predicts affine-invariant depth, then upgrades it to metric depth by estimating the shift and focal length (for which it uses a separate training set with 350k samples). HDN [58] introduced multi-scale depth normalization to improve the prediction details and smoothness further. While this enables the depth estimator to handle images captured with different known cameras, it does not include the true in-the-wild setting, where the camera intrinsics of the test images are unknown. Our method addresses affine-invariant depth estimation but doesn’t focus on compiling an extensive, annotated training dataset. Rather, we utilize the broader image priors in image LDMs and apply fine-tuning.

2.2 Diffusion Models

Denoising Diffusion Probabilistic Models (DDPMs) [18] emerged as a powerful class of generative models. They learn to reverse a diffusion process that progressively degrades images with Gaussian noise so that they can draw samples from the data distribution by applying the reverse process to random noise. This idea was extended to DDIMs [47], which provide a non-Markovian shortcut for the diffusion process. Conditional diffusion models are an extension of DDPMs [18, 47] that ingest additional information on which the output is then conditioned, similar to cGAN [27] and cVAE [46]. Conditioning can take various forms, including text [39], other images [38], or semantic maps [59].

In the realm of text-based image generation, Rombach et al. [36] have trained a diffusion model on the large-scale image and text dataset LAION-5B [44] and demonstrated image synthesis with previously unattainable quality. The cornerstone of their approach is a latent diffusion model (LDM), where the denoising process is run in an efficient latent space, drastically reducing the complexity of the learned mapping. Such models distill internet-scale image sets into model weights, thereby developing a rich scene understanding prior, which we harness for monocular depth estimation.

2.3 Diffusion for Monocular Depth Estimation

Several methods have tried to use DDPMs for metric depth estimation. The DDP approach [20] proposes an architecture to encode the image but decode a depth map and has obtained state-of-the-art results on the KITTI dataset. DiffusionDepth [10] performs diffusion in the latent space, conditioned on image features extracted with a SwinTransformer. DepthGen [42] extends a multi-task diffusion model to metric depth prediction, including handling noisy ground truth. Its successor DDVM [41] emphasizes pretraining on synthetic and real data for enhanced depth estimation. Finally, VPD [62] employs a pretrained Stable Diffusion model as an image feature extractor with additional text input.

Our approach advances beyond these methods, which perform well but only in their specific training domains. We explore the potential of pretrained LDMs for single-image depth estimation across diverse, real-world settings.

2.4 Foundation Models

Vision foundation models (VFMs) are large neural networks trained on internet-scale data. The extreme scaling leads to the emergence of high-level visual understanding, such that the model can then be used as is [50] or fine-tuned to a wide range of downstream tasks with minimal effort [5]. Prompt tuning methods [53, 61, 2] can efficiently adapt VFMs towards dedicated scenarios by designing suitable prompts. Feature adaptation methods [14, 60, 31, 63, 62] can further pivot VFMs towards different tasks. For example, VPD [62] shows the potential to extract features from a pre-trained text-to-image model for (domain-specific) depth estimation. Direct tuning enables more flexible adaptation, not only for few-shot customization scenarios like DreamBooth [37] but also for object detection, as in 3DiffTection [51].

As we show in this paper, Marigold can be interpreted as an instance of this type of tuning, where StableDiffusion plays the role of the foundation model. With as few as 74k synthetic depth samples, we obtain state-of-the-art on multiple datasets of real images. Beyond those datasets, our model exhibits strong in-the-wild performance (cf. Fig. 1).

3 Method

3.1 Generative Formulation

We pose monocular depth estimation as a conditional denoising diffusion generation task and train Marigold to model the conditional distribution $D(0pt~{}|~{}\mathbf{x})$ over depth $0pt\in\mathbb{R}^{W\times H}$, where the condition $\mathbf{x}\ {\in}\ \mathbb{R}^{W\times H\times 3}$ is an RGB image.

In the forward process, which starts at $0pt_{0}:=0pt$ from the conditional distribution, Gaussian noise is gradually added at levels $t\in\{1,...,T\}$ to obtain noisy samples $0pt_{t}$ as

$$0pt_{t}=\sqrt{\bar{\alpha}_{t}}0pt_{0}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon}$$ (1)

where $\bm{\epsilon}\sim\mathcal{N}(0,I),\bar{\alpha}_{t}:=\prod_{s=1}^{t}{1\!-\!\beta_{s}}$, and $\{\beta_{1},\ldots,\beta_{T}\}$ is the variance schedule of a process with $T$ steps. In the reverse process, the conditional denoising model $\bm{\epsilon}_{\theta}(\cdot)$ parameterized with learned parameters $\theta$ gradually removes noise from $0pt_{t}$ to obtain $0pt_{t-1}$.

At training time, parameters $\theta$ are updated by taking a data pair $(\mathbf{x},0pt)$ from the training set, noising $0pt$ with sampled noise $\bm{\epsilon}$ at a random timestep $t$, computing the noise estimate $\hat{\bm{\epsilon}}=\bm{\epsilon}_{\theta}(0pt_{t},\mathbf{x},t)$ and minimizing one of the denoising diffusion objective functions. The canonical standard noise objective $\mathcal{L}$ is given as follows [18]:

$$\mathcal{L}=\mathbb{E}_{0pt_{0},\bm{\epsilon}\sim\mathcal{N}(0,I),t\sim\mathcal{U}(T)}\left\|\bm{\epsilon}-\hat{\bm{\epsilon}}\right\|^{2}_{2}.$$ (2)

At inference time, $0pt:=0pt_{0}$ is reconstructed starting from a normally-distributed variable $0pt_{T}$, by iteratively applying the learned denoiser $\bm{\epsilon}_{\theta}(0pt_{t},\mathbf{x},t)$.

Unlike diffusion models that work directly on the data, latent diffusion models perform diffusion steps in a low-dimensional latent space, offering computational efficiency and suitability for high-resolution image generation [36]. The latent space is constructed in the bottleneck of a variational autoencoder (VAE) trained independently of the denoiser to enable latent space compression and perceptual alignment with the data space. To translate our formulation into the latent space, for a given depth map $0pt$, the corresponding latent code is given by the encoder $\mathcal{E}$: $\mathbf{z}^{(0pt)}=\mathcal{E}(0pt)$. Given a depth latent code, a depth map can be recovered with the decoder $\mathcal{D}$: $\hat{0pt}=\mathcal{D}(\mathbf{z}^{(0pt)})$. The conditioning image $\mathbf{x}$ is also naturally translated into the latent space as $\mathbf{z}^{(\mathbf{x})}=\mathcal{E}(\mathbf{x})$. The denoiser is henceforth trained in the latent space: $\bm{\epsilon}_{\theta}(\mathbf{z}^{(0pt)}_{t},\mathbf{z}^{(\mathbf{x})},t)$. The adapted inference procedure involves one extra step – the decoder $\mathcal{D}$ reconstructing the data $\hat{0pt}$ from the estimated clean latent $\mathbf{z}^{(0pt)}_{0}$: $\hat{0pt}=\mathcal{D}(\mathbf{z}^{(0pt)}_{0})$.

3.2 Network Architecture

Figure 2: Overview of the Marigold fine-tuning protocol. Starting from a pretrained Stable Diffusion, we encode the image $\mathbf{x}$ and depth $0pt$ into the latent space using the original Stable Diffusion VAE. We fine-tune just the U-Net by optimizing the standard diffusion objective relative to the depth latent code. Image conditioning is achieved by concatenating the two latent codes before feeding them into the U-Net. The first layer of the U-Net is modified to accept concatenated latent codes. See details in Sec. 3.2 and Sec. 3.3.

One of our main objectives is training efficiency since diffusion models are often extremely resource-intensive to train. Therefore, we base our model on a pretrained text-to-image LDM (Stable Diffusion v2 [36]), which has learned very good image priors from LAION-5B [44]. With minimal changes to the model components, we turn it into an image-conditioned depth estimator. Fig. 2 contains an overview of the proposed fine-tuning procedure.

Depth encoder and decoder. We take the frozen VAE to encode both the image and its corresponding depth map into a latent space for training our conditional denoiser. Given that the encoder, which is designed for 3-channel (RGB) inputs, receives a single-channel depth map, we replicate the depth map into three channels to simulate an RGB image. At this point, the data range of the depth data plays a significant role in enabling affine-invariance. We discuss our normalization approach in Sec. 3.3. We verified that without any modification of the VAE or the latent space structure, the depth map can be reconstructed from the encoded latent code with a negligible error, i.e., $0pt\approx\mathcal{D}(\mathcal{E}(0pt))$. At inference time, the depth latent code is decoded once at the end of diffusion, and the average of three channels is taken as the predicted depth map.

Adapted denoising U-Net. To implement the conditioning of the latent denoiser $\bm{\epsilon}_{\theta}(\mathbf{z}^{(0pt)}_{t},\mathbf{z}^{(\mathbf{x})},t)$ on input image $\mathbf{x}$, we concatenate the image and depth latent codes into a single input $\mathbf{z}_{t}=\text{cat}(\mathbf{z}^{(0pt)}_{t},\mathbf{z}^{(\mathbf{x})})$ along the feature dimension. The input channels of the latent denoiser are then doubled to accommodate the expanded input $\mathbf{z}_{t}$. To prevent inflation of activations magnitude of the first layer and keep the pre-trained structure as faithfully as possible, we duplicate the weight tensor of the input layer and divide its values by two.

3.3 Fine-Tuning Protocol

Affine-invariant depth normalization. For the ground truth depth maps $0pt$, we implement a linear normalization such that the depth primarily falls in the value range $[-1,1]$, to match the designed input value range of the VAE. Such normalization serves two purposes. First, it is the convention for working with the original Stable Diffusion VAE. Second, it enforces a canonical affine-invariant depth representation independent of the data statistics – any scene must be bounded by near and far planes with extreme depth values. The normalization is achieved through an affine transformation computed as

$$\tilde{0pt}=\left(\frac{0pt-0pt_{2}}{0pt_{98}-0pt_{2}}-0.5\right)\times 2,$$ (3)

where $0pt_{2}$ and $0pt_{98}$ correspond to the $2\%$ and $98\%$ percentiles of individual depth maps. This normalization allows Marigold to focus on pure affine-invariant depth estimation.

Training on synthetic data. Real depth datasets suffer from missing depth values caused by the physical constraints of the capture rig and the physical properties of the sensors. Specifically, the disparity between cameras and reflective surfaces diverting LiDAR laser beams are inevitable sources of ground truth noise and missing pixels [49, 19]. In contrast to prior work that utilized diverse real datasets to achieve generalization [33, 11], we train exclusively with synthetic depth datasets. As with the depth normalization rationale, this decision has two objective reasons. First, synthetic depth is inherently dense and complete, meaning that every pixel has a valid ground truth depth value, allowing us to feed such data into the VAE, which can not handle data with invalid pixels. Second, synthetic depth is the cleanest possible form of depth, which is guaranteed by the rendering pipeline. If our assumption about the possibility of fine-tuning a generalizable depth estimation from a text-to-image LDM is correct, then synthetic depth gives the cleanest set of examples and reduces noise in gradient updates during the short fine-tuning protocol. Thus, the remaining concern is the sufficient diversity or domain gaps between synthetic and real data, which sometimes limits generalization ability. As demonstrated in our experiments, our choice of synthetic datasets leads to impressive zero-shot transfer.

Annealed multi-resolution noise.

Figure 3: Overview of the Marigold inference scheme. Given an input image $\mathbf{x}$, we encode it with the original Stable Diffusion VAE into the latent code $\mathbf{z}^{(\mathbf{x})}$, and concatenate with the depth latent $\mathbf{z}^{(0pt)}_{t}$ before giving it to the modified fine-tuned U-Net on every denoising iteration. After executing the schedule of $T$ steps, the resulting depth latent $\mathbf{z}^{(0pt)}_{0}$ is decoded into an image, whose 3 channels are averaged to get the final estimation $\hat{0}pt$. See Sec. 3.4 for details.

Previous works have explored deviations from the original DDPM formulations, such as non-Gaussian noise [28] or non-Markovian schedule shortcuts [47]. Our proposed setting and the fine-tuning protocol outlined above are permissive to changes to the noise schedule at the fine-tuning stage. We identified a combination of multi-resolution noise [21] and an annealed schedule to converge faster and substantially improve performance over the standard DDPM formulation. The multi-resolution noise is composed by superimposing several random Gaussian noise images of different scales, all upsampled to the U-Net input resolution. The proposed annealed schedule interpolates between the multi-resolution noise at $t=T$ and standard Gaussian noise at $t=0$.

3.4 Inference

Latent diffusion denoising. The overall inference pipeline is presented in Fig. 3. We encode the input image into the latent space, initialize depth latent as standard Gaussian noise, and progressively denoise it with the same schedule as during fine-tuning. We empirically find that initializing with standard Gaussian noise gives better results than with multi-resolution noise, although the model is trained on the latter. We follow DDIM’s [47] approach to perform non-Markovian sampling with re-spaced steps for accelerated inference. The final depth map is decoded from the latent code using the VAE decoder and postprocessed by averaging channels.

Test-time ensembling. The stochastic nature of the inference pipeline leads to varying predictions depending on the initialization noise in $\mathbf{z}^{(0pt)}_{T}$. Capitalizing on that, we propose the following test-time ensembling scheme, capable of combining multiple inference passes over the same input. For each input sample, we can run inference $N$ times. To aggregate these affine-invariant depth predictions $\{\mathbf{\hat{d}}_{1},\ldots,\mathbf{\hat{d}}_{N}\}$, we jointly estimate the corresponding scale $\hat{s}_{i}$ and shift $\hat{t}_{i}$, relative to some canonical scale and range, in an iterative manner. The proposed objective minimizes the distances between each pair of scaled and shifted predictions $(\mathbf{\hat{d^{\prime}}}_{i},\mathbf{\hat{d^{\prime}}}_{j})$, where $\mathbf{\hat{d^{\prime}}}=\mathbf{\hat{d}}\times\hat{s}+\hat{t}$. In each optimization step, we calculate the merged depth map $\mathbf{m}$ by the taking pixel-wise median $\mathbf{m}(x,y)=\text{median}(\mathbf{\hat{d^{\prime}}}_{1}(x,y),\ldots,\mathbf{\hat{d^{\prime}}}_{N}(x,y))$. An extra regularization term $\mathcal{R}=|\!\min(\mathbf{m})|+|1-\max(\mathbf{m})|$, is added to prevent collapse to the trivial solution and enforce the unit scale of $\mathbf{m}$. Thus, the objective function can be written as follows:

$$\min_{\begin{subarray}{c}s_{1},\ldots,s_{N}\\ t_{1},\ldots,t_{N}\end{subarray}}\Bigg{(}\sqrt{\frac{1}{b}\sum_{i=1}^{N-1}\sum_{j=i+1}^{N}\|\mathbf{\hat{d^{\prime}}}_{i}-\mathbf{\hat{d^{\prime}}}_{j}\|_{2}^{2}}+\lambda\mathcal{R}\Bigg{)}$$ (4)

where the binominal coefficient $b=\binom{N}{2}$ represents the number of possible combinations of image pairs from $N$ images. After the iterative optimization for spatial alignment, the merged depth $\mathbf{m}$ is taken as our ensembled prediction. Note that this ensembling step requires no ground truth for aligning independent predictions. This scheme enables a flexible trade-off between computation efficiency and prediction quality by choosing $N$ accordingly.

4 Experiments

4.1 Implementation

We implement Marigold using PyTorch and utilize Stable Diffusion v2 [36] as our backbone, following the original pre-training setup with a v-objective [40]. We disable text conditioning and perform steps outlined in Sec. 3.2. During training, we apply the DDPM noise scheduler [18] with 1000 diffusion steps. At inference time, we apply DDIM scheduler [47] and only sample 50 steps. For the final prediction, we aggregate results from 10 inference runs with varying starting noise. Training our method takes 18K iterations using a batch size of 32. To fit one GPU, we accumulate gradients for 16 steps. We use the Adam optimizer with a learning rate of $3\cdot 10^{-5}$. Additionally, we apply random horizontal flipping augmentation to the training data. Training our method to convergence takes approximately 2.5 days on a single Nvidia RTX 4090 GPU card.

Table 1: Quantitative comparison of Marigold with SOTA affine-invariant depth estimators on several zero-shot benchmarks. All metrics^† are presented in percentage terms; bold numbers are the best, underscored second best. Our method outperforms other methods on both indoor and outdoor scenes in most cases, without ever seeing a real depth sample.

Method	# Training samples			NYUv2			KITTI			ETH3D			ScanNet			DIODE			Avg. Rank
	Real	Synthetic		AbsRel↓	$\delta$1↑		AbsRel↓	$\delta$1↑		AbsRel↓	$\delta$1↑		AbsRel↓	$\delta$1↑		AbsRel↓	$\delta$1↑
DiverseDepth [54]	320K	—		11.7	87.5		19.0	70.4		22.8	69.4		10.9	88.2		37.6	63.1		6.6
MiDaS [33]	2M	—		11.1	88.5		23.6	63.0		18.4	75.2		12.1	84.6		33.2	71.5		6.3
LeReS [55]	300K	54K		9.0	91.6		14.9	78.4		17.1	77.7		9.1	91.7		27.1	76.6		4.3
Omnidata [11]	11.9M	310K		7.4	94.5		14.9	83.5		16.6	77.8		7.5	93.6		33.9	74.2		3.8
HDN [58]	300K	—		6.9	94.8		11.5	86.7		12.1	83.3		8.0	93.9		24.6	78.0		2.4
DPT [34]	1.2M	188K		9.8	90.3		10.0	90.1		7.8	94.6		8.2	93.4		18.2	75.8		3.1
Marigold (ours)	—	74K		5.5	96.4		9.9	91.6		6.5	96.0		6.4	95.1		30.8	77.3		1.4

• Most baselines are sourced from Metric3D [56], except the ScanNet benchmark. For ScanNet, Metric3D used a different random split that is not publicly accessible. Therefore, we re-ran baseline methods on our split. We additionally took numbers from Metric3D for HDN [58] on ScanNet benchmark due to unavailable source code.

4.2 Evaluation

Training datasets. We train Marigold on two synthetic datasets covering both indoor and outdoor scenes. Hypersim [35] is a photorealistic dataset with 461 indoor scenes. We use the official split with around 54K samples from 365 scenes for training. Incomplete samples are filtered out. RGB images and depth maps are resized to $480\times 640$ size. Depth is normalized with the dataset statistics. Additionally, we transform the original distances relative to the focal point into conventional depth values relative to the focal plane. The second dataset, Virtual KITTI [6] is a synthetic street-scene dataset featuring 5 scenes under varying conditions like weather or camera perspectives. Four scenes containing a total of around 20K samples are used for training. We crop the images to the KITTI benchmark resolution [15] and set the far plane to 80 meters.

Evaluation datasets. We evaluate Marigold on 5 real datasets that are not seen during training. NYUv2 [29] and ScanNet [9] are both indoor scene datasets captured with an RGB-D Kinect sensor. For NYUv2, we utilize the designated test split, comprising a total of 654 images. In the case of the ScanNet dataset, we randomly sampled 800 images from the 312 official validation scenes for testing. KITTI [15] is a street-scene dataset with sparse metric depth captured by a LiDAR sensor. We employ the Eigen test split [12] made of 652 images. ETH3D [43] and DIODE [48] are two high-resolution datasets, both featuring depth maps derived from LiDAR sensor measurements. For ETH3D, we incorporate all 454 samples with available ground truth depth maps. For DIODE, we use the entire validation split, which encompasses 325 indoor samples and 446 outdoor samples.

Evaluation protocol. Following the protocol of affine-invariant depth evaluation [33], we first align the estimated merged prediction $\mathbf{m}$ to the ground truth $0pt$ with the least squares fitting. This step gives us the absolute aligned depth map $\mathbf{a}=\mathbf{m}\times s+t$ in the same units as the ground truth. Next, we apply two widely recognized metrics [56, 33, 34, 55] for assessing quality of depth estimation. The first is Absolute Mean Relative Error (AbsRel), calculated as: $\frac{1}{M}\sum_{i=1}^{M}{|\mathbf{a}_{i}-0pt_{i}|}/{0pt_{i}}$, where $M$ is the total number of pixels. The second metric, $\delta 1$ accuracy, measures the proportion of pixels satisfying $\max({\mathbf{a}_{i}}/{0pt_{i}},{0pt_{i}}/{\mathbf{a}_{i}})<1.25$.

Comparison with other methods. We compare Marigold to six baselines, each claiming zero-shot generalization. DiverseDepth [54], LeReS [55] and HDN [58] estimate affine-invariant depth maps, while MiDaS [33], DPT [34], and Omnidata [11] produce affine-invariant disparities. As shown in Tab. 1, Marigold outperforms prior art in most cases and secures the highest overall ranking. Despite being trained solely on synthetic depth datasets, the model can well generalize to a wide range of real scenes. This successful adaptation of diffusion-based image generation models toward depth estimation confirms our initial hypothesis that a comprehensive representation of the visual world is the cornerstone of monocular depth estimation. It also shows that our fine-tuning protocol was successful in adapting Stable Diffusion for this task without unlearning such visual priors.

For a visual assessment, we present qualitative comparison in Fig. 4. Additionally, in Fig. 5, we provide 3D visualizations of reconstructed surface normals. Marigold not only correctly captures the scene layout, such as the spatial relationships between walls and furniture in the first example in Fig. 5, but also captures fine-grained details, as indicated by the arrows in Fig. 4. Moreover, the reconstruction of flat surfaces, especially walls, is significantly better (see Fig. 4). Furthermore, our method effectively models common shapes and their layouts, once again aligning with our expectations regarding the generative prior.

	Input RGB Image	MiDaS	Omnidata	DPT	Marigold (ours)	Ground Truth
NYUv2
KITTI
ETH3D
Scannet
DIODE

Figure 4: Qualitative comparison (depth) of monocular depth estimation methods across different datasets. Marigold excels at capturing thin structures (e.g., chair legs) and preserving overall layout of the scene (e.g., walls in ETH3D example and chairs in DIODE example).

	Input RGB Image	MiDaS	Omnidata	DPT	Marigold (ours)	Ground Truth
NYUv2
ScanNet
DIODE

Figure 5: Qualitative comparison (unprojected, colored as normals) of monocular depth estimation methods across different datasets. Marigold stands out for its superior reconstruction of flat surfaces and detailed structures.

4.3 Ablation Studies

Two zero-shot validation sets are selected for the ablation studies – the official training split of NYUv2 [29], consisting of 785 samples, and a randomly selected subset of 800 images from the KITTI Eigen [12] training split. Refer to supplementary sections for extra ablations and discussion.

Figure 6: Ablation of ensemble size. We observe a monotonic improvement with the growth of ensemble size. This improvement starts to diminish after 10 predictions per sample.

Table 2: Ablation of training noise. Multi-resolution noise improves over Gaussian noise; annealing yields further improvement.

Multi-res. noise	Annealed		NYUv2		KITTI
			AbsRel↓	$\delta$1↑	AbsRel↓	$\delta$1↑
✗	-		7.7	93.4	14.2	82.1
✓	✗		5.8	96.1	12.1	87.1
✓	✓		5.6	96.5	11.3	88.7

Training noise. We investigate the impact of three types of noise during the training phase. As shown in Tab. 2, training with multi-resolution noise significantly improves the depth prediction accuracy over using standard Gaussian noise. Furthermore, the gradual annealing of multi-resolution noise yields an additional improvement. We also noticed that training with multi-resolution noise leads to more consistent predictions given different initial noise at inference time and annealing further enhances this consistency.

Training data domain. To better understand the impact of the synthetic datasets used for our fine-tuning protocol, we ablate on a photorealistic street-scene Virtual KITTI [6], and a more diverse and higher-quality indoor dataset Hypersim [35]. The results are shown in Tab. 3. When fine-tuned on a single synthetic dataset, the pretrained LDM can already be adapted for monocular depth estimation to a certain degree, while the more diverse and photorealistic data leads to better performance on both indoor and outdoor scenes. Interestingly, adding additional training data from a different domain not only improves the performance on the new domain but also brings improvements in the original domain.

Test-time ensembling. We test the effectiveness of the proposed test-time ensembling scheme by aggregating various numbers of predictions. As shown in Fig. 6, a single prediction of Marigold already yields reasonably good results. Ensembling 10 predictions reduces the absolute relative error on NYUv2 by ${\sim}8\%$ and ensembling 20 predictions brings an improvement of ${\sim}9.5\%$. It has been observed as a systematic effect that the performance is constantly improved as the number of predictions increases, while the marginal improvement diminishes with more than 10 predictions.

Figure 7: Ablation of denoising steps. The performance improves as the number of denoising steps increases, while we observe saturation after 10 steps.

Table 3: Ablation of training datasets. Hypersim [35] delivers strong results; Virtual KITTI [6] improves outdoor performance.

Hypersim	Virtual KITTI		NYUv2		KITTI
			AbsRel↓	$\delta$1↑	AbsRel↓	$\delta$1↑
✗	✓		13.9	83.4	15.4	79.3
✓	✗		5.7	96.3	13.7	82.5
✓	✓		5.6	96.5	11.3	88.7

Number of denoising steps. We evaluate the effect of the re-spaced inference denoising steps driven by the DDIM scheduler [47]. The results are shown in Fig. 7. Although trained with 1000 DDPM steps, the choice of 50 steps is sufficient to produce accurate results during inference. As expected, we obtain better results when using more denoising steps. We observe that the elbow point of marginal returns given more denoising steps depends on the dataset but is always under 10 steps. This implies that one can further reduce the denoising steps to 10 or even less to gain efficiency while keeping comparable performance. Interestingly, this threshold is smaller than what is usually required for diffusion-based image generators [47, 36], i.e., 50 steps.

5 Conclusion

We presented Marigold, a fine-tuning protocol for Stable Diffusion and a model for state-of-the-art affine-invariant depth estimation. Evaluation results of our method confirm the importance of rich scene understanding prior, which we harness from the pretrained text-to-image latent diffusion model. Future research directions could find ways to counter the limitations of the current approach, such as handling a higher depth range of the scenes, possibly including infinite depth prediction, as well as more efficient inference and automatic selection of denoising steps.

Appendix

In this supplementary material, we provide additional implementation details in Appendix A and present additional quantitative and qualitative results in Appendix B and Appendix C, respectively.

Appendix A Implementation Details

A.1 Mixed Dataset Training

We train on two synthetic datasets, Hypersim [35] and Virtual KITTI [6], whose images have different resolutions and aspect ratios. For each batch, we probabilistically choose the dataset and then draw samples from it. We ablate the Bernoulli parameter of dataset sampling in Sec. B.4.

A.2 Annealed Multi-Resolution Noise

In the standard multi-resolution noise, multiple Gaussian noise images are sampled to form a pyramid of resolutions and then subsequently combined by upsampling, weighted averaging, and renormalization. The weight for the $i$-th pyramid level is computed as $s^{i}$, where $0<s<1$ is a strength of influence of lower-resolution noise. To bring such noise closer to the Gaussian used in the original DDPM formulation, we propose to anneal the weight of levels $i>0$ based on the diffusion schedule. Specifically, we assign the $i$-th level at timestep $t$ the weight $(st/T)^{i}$, where $T$ is the total number of diffusion steps. Thus, a smaller weight is given to lower-resolution levels at timesteps closer to the noise-free end of the schedule. In addition to the ablation study in the main paper, we further demonstrate the effectiveness of annealing and other noise settings in Sec. B.3.

A.3 Alignment with Ground Truth Depth

Following the established evaluation protocol [33], we use least squares fitting over pixels with valid ground truth values to compute the scale and shift factors of the affine-invariant predictions. Note that, while some methods predict affine-invariant disparities [33, 34, 11], others (including ours) predict affine-invariant depth values [54, 55, 58]. We apply least squares fitting accordingly, i.e. the disparities are aligned to the inverse ground truth depth.

A.4 Visualization in 3D

We compute the scale and shift scalars between the prediction and ground truth. Subsequently, we unproject pixels into the metric 3D space using the camera intrinsics. We manually estimate the scale, shift, and intrinsics of “in-the-wild” samples, where ground truth and camera intrinsics are unavailable. For some samples, camera intrinsics can also be extracted from the EXIF metadata. To visualize normals, we perform least squares plane fitting at each position, considering a neighborhood area of $3\times 3$ pixels around it.

Appendix B Experimental Results

B.1 Stable Diffusion VAE with Depth

To assess how well the pre-trained image variational autoencoder of Stable Diffusion [36] works with depth maps, we tested it with 800 samples from the Hypersim [35] training set. To this end, each sample is normalized to the operational range of VAE as explained in the main paper, and replicated three times to accommodate the RGB interface. Upon decoding the latents, the reconstructed depth map is derived by averaging the three RGB channels. Over the chosen set of depth maps, the Mean Absolute Error (MAE) of reconstructions is $0.0095\pm 0.0091$, which is safely below the current state-of-the-art depth estimation errors.

B.2 Consistency of Channels After VAE Decoder

To further understand the suitability of the Stable Diffusion latent space for depth representation, we evaluate the agreement of depth channels obtained from the VAE decoder during inference. We validate with the training split of NYUv2 [29] and a subsampled Eigen training split [12] of the KITTI dataset [15]. As shown in Tab. S1, the channel-wise discrepancy resulting from decoding depth from the latent space is small relative to the value range of the decoder output, i.e., $[-1,1]$. This could be related to the ability of VAE to represent gray-scale RGB images.

Table S1: Consistency of channels after VAE decoder. The reported numbers are averaged over the respective datasets.

	std	$\text{max}-\text{min}$
NYU	0.0027	0.0062
KITTI	0.0022	0.0052

B.3 Prediction Variance and Training Noise

Since Marigold is a generative model, the predictions vary depending on the initial noise starting the diffusion process. We evaluate the consistency of predictions of three models, trained differently, i.e., with Gaussian noise, multi-resolution noise, and annealed multi-resolution noise. We train with two synthetic datasets and validate with the training split of NYUv2 [29] and a subsampled Eigen training split [12] of the KITTI dataset [15]. Specifically, we perform inference 10 times for each sample and compute pixel-wise statistics over the resulting depth predictions. Subsequently, we aggregate these statistics across entire datasets and report them in Tab. S2. As seen from the values, training with the multi-resolution noises increases the prediction consistency at inference, and the annealed version brings further improvement. Fig. S1 demonstrates predictions for a single sample with three models and varying starting noise.

Table S2: Pixel-wise consistency of depth predictions made by models trained with three different noise types. The reported numbers are averaged over entire datasets, wherein each sample was processed 10 times, starting from a new noise sample.

Multi-res. noise	Annealed		NYUv2		KITTI
			std	$\text{max}-\text{min}$	std	$\text{max}-\text{min}$
✗	✗		0.086	0.260	0.050	0.152
✓	✗		0.037	0.117	0.030	0.094
✓	✓		0.033	0.106	0.025	0.079

Figure S1: Example of predictions on the same input by models trained with (top-down) Gaussian, multi-resolution, and annealed multi-resolution noise. The last row exhibits the least variance.

B.4 Ratio of Mixed Training Datasets

To further investigate the impact of the synthetic datasets used in our fine-tuning protocol, we ablate the mixing ratio of the datasets, discussed in Sec. A.1. We train with two synthetic datasets, Hypersim [35] and Virtual KITTI [6], and validate with the training split of NYUv2 [29] and a subsampled Eigen training split [12] of the KITTI dataset [15]. As shown in Tab. S3, training with a mixture of these two synthetic datasets yields better results on both indoor and outdoor real datasets, than training with a single synthetic dataset. Interestingly, based on the higher-quality indoor dataset, Hypersim [35], adding a small portion (5%) of Virtual KITTI [6], a street-view dataset, can already increase the performance on the outdoor dataset. We find a sweet spot at around 10% where the performance is improved on both indoor and outdoor scenes. When the ratio of Virtual KITTI keeps increasing, the overall performance is impaired. This is likely caused by the varying scene diversity and rendering quality of these two datasets.

Table S3: Ablation study of the training dataset mixing strategy. Our method trained with only Hypersim delivers strong results. Outdoor performance is further enhanced with a small portion of Virtual KITTI. The zero-shot transfer is attained at 10% ratio.

Hypersim	Virtual KITTI		NYUv2		KITTI
			AbsRel↓	$\delta$1↑	AbsRel↓	$\delta$1↑
100%	0%		5.7	96.3	13.7	82.5
95%	5%		5.8	96.2	11.1	88.8
90%	10%		5.6	96.5	11.3	88.7
50%	50%		6.0	96.0	12.8	85.5
0%	100%		13.9	83.4	15.4	79.3

Appendix C Qualitative Comparisons

C.1 In-the-Wild

We present the gallery of “in-the-wild” images and corresponding predictions in Fig. S2. The input images are taken in daily life or downloaded from the internet. Our method, Marigold, predicts accurate depth maps, exhibiting better overall layout and fine details. We show the final predictions for each method, that is, depth for Marigold and LeReS, and disparity for MiDaS.

C.2 Test Datasets

We show additional qualitative comparisons with our competitors [54, 33, 55, 34, 11], on 5 test datasets [29, 15, 43, 9, 48]. The depth maps are visualized in Fig. S3, and the normal maps can be found in Fig. S4. Marigold excels at capturing fine scene details and reflecting the global scene layout.

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Input RGB Image	Marigold (ours, depth)	LeReS (depth)	MiDaS (disparity)

Figure S2: Qualitative comparison on in-the-wild scenes. Marigold and LeReS predict depth (with red indicating closer and blue indicating farther distances), while MiDaS predicts disparity (with yellow signifying closer and purple signifying farther distances).

NYUv2 [29]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

NYUv2 [29]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
KITTI [15]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

KITTI [15]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

KITTI [15]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
ETH3D [43]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

ETH3D [43]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

ScanNet [9]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

ScanNet [9]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
DIODE [48]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

DIODE [48]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

Figure S3: Qualitative comparison (depth) of monocular depth estimation methods across different datasets. Predictions are aligned to ground truth. For every sample, the color coding is consistent across all depth maps.

NYUv2 [29]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

ETH3D [43]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

ScanNet [9]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata
DIODE [48]	Input RGB Image	DiverseDepth	MiDaS	LeReS
	Ground Truth	Marigold (ours)	DPT	Omnidata

Figure S4: Qualitative comparison (unprojected, colored as normals) of monocular depth estimation methods across different datasets. Ground truth normals are derived from the ground truth depth maps.