Learning Temporally Consistent Video Depth from Video Diffusion Priors

Abstract

This work addresses the challenge of video depth estimation, which expects not only per-frame accuracy but, more importantly, cross-frame consistency. Instead of directly developing a depth estimator from scratch, we reformulate the prediction task into a conditional generation problem. This allows us to leverage the prior knowledge embedded in existing video generation models, thereby reducing learning difficulty and enhancing generalizability. Concretely, we study how to tame the public Stable Video Diffusion (SVD) to predict reliable depth from input videos using a mixture of image depth and video depth datasets. We empirically confirm that a procedural training strategy — first optimizing the spatial layers of SVD and then optimizing the temporal layers while keeping the spatial layers frozen — yields the best results in terms of both spatial accuracy and temporal consistency. We further examine the sliding window strategy for inference on arbitrarily long videos. Our observations indicate a trade-off between efficiency and performance, with a one-frame overlap already producing favorable results. Extensive experimental results demonstrate the superiority of our approach, termed ChronoDepth, over existing alternatives, particularly in terms of the temporal consistency of the estimated depth. Additionally, we highlight the benefits of more consistent video depth in two practical applications: depth-conditioned video generation and novel view synthesis. Our project page is available at https://jhaoshao.github.io/ChronoDepth/.

1 Introduction

Monocular video depth estimation stands as a foundational challenge within the realm of computer vision, boasting diverse applications across robotics, autonomous driving, animation, virtual reality, and beyond. While recent strides have notably enhanced the spatial precision of depth prediction through single-image depth models [55, 77, 17, 79, 54, 73, 36, 20, 23, 72], achieving temporal consistency over a video remains an ongoing pursuit. Temporal consistency, vital for eliminating flickering artifacts between successive frames caused by single-frame scale ambiguity, remains a core objective in the domain of video depth estimation. Existing methodologies [47, 40, 82] predominantly employ test-time training (TTT) paradigms, wherein a single-image depth model undergoes fine-tuning on the testing video, leveraging geometry constraints and pose estimation for inference. However, this approach heavily hinges on precise camera poses and lacks generalizability. Despite concerted efforts to enforce temporal consistency through learning-based models, prior methods [70, 45, 80, 68, 12] have often faltered, exhibiting inferior performance metrics. While certain techniques [69] demonstrate promising temporal coherence, their spatial accuracy remains suboptimal, failing to meet desired standards. Thus, the quest persists for a unified model capable of seamlessly marrying spatial accuracy with temporal consistency, a pursuit that promises to advance the frontiers of monocular video depth estimation.

Video generative models have witnessed significant growth recently, sharing similar targets of achieving high spatial quality with temporal consistency. Recent advancements in video generation [9, 8] have yielded models capable of producing visually realistic and temporally consistent content. These achievements often stem from the integration of temporal layers, which incorporate self-attention mechanisms and convolutional operations between frames within conventional image generation frameworks. Through this streamlined approach, these models effectively synthesize video content that exhibits both realism and temporal coherence, marking a notable progression in video generation technology.

In this work, inspired by the recent attempts leveraging the priors of image generative models for single-frame perceptual tasks [33, 16, 58, 84, 15, 36, 20, 23, 72], we investigate the potential of pre-trained video generative models to serve as foundational models for video depth estimation. Considering that fine-tuning a video diffusion model for discriminative depth estimation has not been explored in the literature, we delve into optimizing the fine-tuning protocols and inference strategies to achieve both spatial accuracy and temporal consistency. For training, we conduct thorough comparisons between various protocols, empirically finding an effective training strategy fully exploring image and video depth datasets. In particular, we fine-tune SVD by combining image and video depth datasets, using randomly sampled clip lengths from the latter. Instead of fine-tuning the full network, we observe the sequential training of spatial and temporal layers to be more effective, where the spatial layers are first trained and kept frozen during the training of the temporal layers. For inference, we use a sliding windows strategy, using previously predicted depth frames to guide the prediction of subsequent frames, yielding consistent depth estimation over videos of unlimited length. We observe that using only a single frame overlap between two consecutive videos yields good temporal consistency, effectively reaching a balance between efficiency and performance. Through training on synthetic image and video datasets, we conduct quantitative and qualitative evaluations on real-world benchmarks. Experimental results show that our method achieves state-of-the-art temporal consistency on video depth estimation, with comparable spatial accuracy with state-of-the-art single-image depth estimation methods. We further demonstrate that our temporally-consistent depth estimation provides better support than baselines for downstream applications, including depth-conditioned video generation and novel view synthesis.

2 Related Work

2.1 Monocular Depth Estimation

Estimating depth from a single image is a ubiquitous task in computer vision, addressed through either discriminative or generative methodologies.

Discriminative depth estimation. These approaches are trained end-to-end to regress depth, according to two alternative categories: metric versus relative depth estimation. Early attempts focused on the former category [18], yet were limited to single scenarios – i.e., training specialized models for driving environments [22] or indoor scenes [60]. Among these, some frameworks used ordinal regression [19], local planar guidance [41], adaptive bins [6] or fully-connected CRFs [78]. In pursuit of achieving generalizability across different environments [55], the community recently shifted towards training models estimating relative depth, through the use of affine-invariant loss functions [55, 54] to be robust against different scale and shift factors across diverse datasets. On this track, the following efforts focused on recovering the real 3D shape of scenes [77], exploiting surface normals as supervision [75], improving high-frequency details [48, 79] or exploiting procedural data generation [17]. To combine the best of the two worlds, new attempts to learn generalizable, metric depth estimation have emerged lately [7, 43], most of them by explicitly handling camera parameters through canonical transformations [76, 29] or as direct prompts to the model [25, 52].

Generative depth estimation. Eventually, some methodologies have embraced the utilization of pre-trained generative models for monocular depth estimation. Among them, some exploited Low-Rank Adaptation (LoRA) [15] or started from self-supervised pre-training [58], while others re-purposed Latent Diffusion Models (LDM) by fine-tuning the pre-trained UNet [36] to denoise depth maps. Further advances of this latter strategy jointly tackled depth estimation and surface normal prediction [20], or exploited Flow Matching [23] for higher efficiency. Despite achieving remarkable performance and capturing intricate details in complex scenarios, these methods often struggle to maintain geometric consistency when extrapolated to 3D space and exhibit subpar temporal consistency when estimating depth across sequences of frames. To address these challenges, we take a further step forward on this track by exploiting a pre-trained video diffusion model [8] as a robust visual prior in place of a single-image one, while designing an efficient fine-tuning protocol to deal with the high complexity it demands.

2.2 Video Depth Estimation

In addition to spatial accuracy, achieving temporal consistency is a fundamental goal when predicting depth on a video. This entails eliminating flickering effects between consecutive frames, possibly caused by scale inconsistencies. Some approaches hinge on estimating the poses of any frame in the video and using them to build cost volumes [70, 45, 24] or running test-time training [47, 40, 82], with both heavily depending on the accuracy of the poses and the latter lacking generalizability as it overfits a single video. Others deploy recurrent networks [80, 64], while most recent works exploit attention mechanisms [68, 12], yet with sub-optimal results compared to state-of-the-art single-image depth predictors. Finally, NVDS [69] introduces a stabilization network so as to conduct temporal refinement to single-frame results from an off-the-shelf depth predictor, which could harm spatial accuracy.

2.3 Diffusion Models

Image Diffusion Models (IDMs) by Sohl-Dickstein et al. [61] conquered the main stage for image generation tasks [14, 39] at the expense of GANs. Further developments aimed at improving both generation conditioning [1, 50] and computational efficiency [57], with Latent Diffusion Models (LDMs) [57] notably emerging as a solution for the latter. Among conditioning techniques, the use of cross-attention [3, 10, 21, 26, 35, 38, 50, 51, 53] and the encoding of segmentation masks into tokens [2, 21] stand as the most popular, with additional schemes being proposed to enable the generation of visual data conditioned by diverse factors such as text, images, semantic maps, sketches, and other representations [4, 5, 30, 49, 81, 66]. Lately, LDMs have been extended for video generation [28, 8, 13, 32, 83, 71], focusing on obtaining consistent content over time thanks to the integration of temporal layers, incorporating self-attention mechanisms and convolutions between frames within conventional image generation frameworks.

While recent efforts exploited image diffusion models for single-image depth estimation [15, 58, 36, 20, 23], ours is the first effort to tame cutting-edge video diffusion models for consistent depth estimation on monocular videos.

3 Method

In this section, we introduce ChronoDepth, a consistent video depth estimator derived from a video foundation model, specifically from Stable Video Diffusion (SVD) [8].

Given a consecutive monocular video $\mathbf{x}$, our goal is to generate spatial-accurate and temporal-consistent video depth $\mathbf{d}$. Firstly, we reformulate this problem as a conditional denoising diffusion generation task (see Sec. 3.1). Secondly, we perform a comprehensive empirical analysis of fine-tuning protocols, aiming to discover the best practices for taming video foundation model into consistent depth estimator (see Sec. 3.2). Lastly, we propose an efficient and effective inference approach and demonstrate it to outperform other inference approaches (see Sec. 3.3). Fig. 1 contains an overview of ChronoDepth.

3.1 Diffusion Formulation

In order to align with the video foundation model, SVD [8], we reformulate monocular video depth estimation as a continuous-time denoising diffusion [62, 34] generation task conditioned on the RGB video. The diffusion model consists of a stochastic forward pass to inject $\sigma^{2}$-variance Gaussian noise into the input video depth and a reverse process to remove noise with a learnable denoiser $D_{\mathbf{\theta}}$. Following SVD, our diffusion model is defined in a latent space of lower spatial dimension for better computation efficiency, where a variational autoencoder (VAE) consists of an encoder $\mathcal{E}$ and a decoder $\mathcal{D}$ is used to compress the original signals. In order to repurpose the VAE of SVD, which accepts a 3-channel (RGB) input, we replicate the depth map into three channels to mimic an RGB image. Subsequently, during inference, we decode and calculate the average of these three channels, which serves as our predicted depth map following [36].

Training. During training, the RGB video clip $\mathbf{x}\in\mathbf{R}^{T\times W\times H\times 3}$ and the corresponding depth $0pt\in\mathbf{R}^{T\times W\times H}$ are first encoded into the latent space with the VAE encoder: $\mathbf{z}^{(\mathbf{x})}=\mathcal{E}(\mathbf{x}),\mathbf{z}^{(0pt)}=\mathcal{E}(0pt)$. For each training step, we sample a Gaussian noise $\mathbf{n}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{M}I)$ and add it to the depth latent $\mathbf{z}^{(0pt)}_{0}$ to obtain $\mathbf{z}^{(0pt)}_{M}$ as

$$\mathbf{z}^{(0pt)}_{M}=\mathbf{z}^{(0pt)}+\mathbf{n},\quad\mathbf{n}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{M}I)$$ (1)

where $\log\sigma_{M}\sim\mathcal{N}(P_{mean},P_{std}^{2})$ with $P_{mean}=0.7$ and $P_{std}=1.6$ [34]. For simplicity, we denote $\sigma_{M}$ as $\sigma$ throughout the remainder of training part. In the reverse process, diffusion model denoises $\mathbf{z}^{(0pt)}_{M}$ towards clean $\mathbf{z}^{(0pt)}$ with a learnable denoiser $D_{\mathbf{\theta}}$, which is trained via denoising score matching (DSM) [65]

$$\mathcal{L}=\mathbb{E}_{{(\mathbf{z}^{(0pt)},\mathbf{z}^{(\mathbf{x})})\sim p_{data}(\mathbf{z}^{(0pt)},\mathbf{z}^{(\mathbf{x})}),(\sigma,\mathbf{n})\sim p(\sigma,\mathbf{n})}}\left[\lambda_{\sigma}\|D_{\theta}(\mathbf{z}^{(0pt)}_{M};\sigma,\mathbf{z}^{(\mathbf{x})})-\mathbf{z}^{(0pt)}\|_{2}^{2}\right],$$ (2)

where weighting function $\lambda(\sigma)=(1+\sigma^{2})\sigma^{-2}$. In this work, we follow the EDM-preconditioning framework [34], parameterizing the learnable denoiser $D_{\mathbf{\theta}}$ as

$$D_{\mathbf{\theta}}(\mathbf{z}^{(0pt)}_{M};\sigma,\mathbf{z}^{(\mathbf{x})})=\frac{1}{\sigma^{2}+1}\mathbf{z}^{(0pt)}_{M}-\frac{\sigma}{\sqrt{\sigma^{2}+1}}F_{\mathbf{\theta}}(\mathbf{z}^{(0pt)}_{M}/{\sqrt{\sigma^{2}+1}};0.25log\sigma,\mathbf{z}^{(\mathbf{x})}),$$ (3)

where $F_{\mathbf{\theta}}$ is a UNet to be trained in our case. The condition, RGB video latent $\mathbf{z}^{(\mathbf{x})}$, in this formulation is introduced via concatenation with depth video latent on feature dimension $\mathbf{z}_{M}=concat(\mathbf{z}^{(0pt)}_{M}/\sqrt{\sigma^{2}+1},\mathbf{z}^{(\mathbf{x})})$.

Inference. During inference time, $\mathbf{z}^{(0pt)}_{0}$ is restored from a randomly-sampled Gaussian noise $\mathbf{z}^{(0pt)}_{N}\sim\mathcal{N}(\mathbf{0},\sigma^{2}_{N}I)$ conditioning on given RGB video via iteratively applying the denoising process with trained denoiser $D_{\mathbf{\theta}}$

$$\mathbf{z}^{(0pt)}_{i}=\mathbf{z}^{(0pt)}_{i+1}+\frac{\mathbf{z}_{i+1}-D_{\mathbf{\theta}}(\mathbf{z}^{(0pt)}_{i+1};\sigma_{i+1},\mathbf{z}^{(\mathbf{x})})}{\sigma_{i+1}}(\sigma_{i}-\sigma_{i+1}),\ i<N$$ (4)

where {$\sigma_{0},...,\sigma_{N}$} is the fixed variance schedule of a denoising process with $N$ steps. $0pt_{0}$ could be obtained with decoder $0pt_{0}=\mathcal{D}(\mathbf{z}^{(0pt)}_{0})$.

Figure 1: Training pipeline. We add an RGB video conditioning branch to a pre-trained video diffusion model, Stable Video Diffusion, and fine-tune it for consistent depth estimation. Both the RGB and depth videos are projected to a lower-dimensional latent space using a pre-trained encoder. The video depth estimator is trained via denoising score matching (DSM). Our training involves two stages: first, we train the spatial layers with single-frame depths; then, we freeze the spatial layers and train the temporal layers using clips of randomly sampled lengths. This sequential spatial-temporal fine-tuning approach yields better performance than training the full network.

3.2 Fine-Tuning Protocol

In contrast to image generative models, video generative models incorporate both spatial and temporal dimensions. The central question thereby becomes how to effectively fine-tune these pre-trained video generative models to achieve a satisfying level of spatial accuracy and temporal coherence in geometry. In response to this, we have conducted comprehensive analyses to ascertain the best practices for taming our foundational video model into a consistent depth estimator. Note that the VAE for compressing the original pixel space remains frozen.

Without v.s. With Single-Frame Datasets. We first investigate the impact of training data for taming SVD into a video depth estimator. In particular, we investigate whether it is beneficial to use single-frame datasets for supervision, in addition to video depth datasets. Thanks to the fact that single-frame depth datasets are easier to obtain and are usually more diverse, our experimental results suggest that jointly using single-frame and multi-frame depth datasets is crucial to achieving good spatial and temporal accuracy. Therefore, we keep the single-frame dataset as a part of our supervision throughout the full training process, in contrast to SVD [8] that single-view image datasets [59] are only used in the first pre-training stage. Note that the total number of training images of the video depth datasets is more than that of the image depth dataset, yet the number of scenes is less, indicating that scene diversity plays an important role in depth estimation.

Fixed v.s. Randomly Sampled Clip Length. Next, we ablate whether the length of the video clips $T$ should be fixed or randomly sampled during the training process. Both options are naturally supported by design, thanks to the spatial layers interpreting the video as a batch of independent images and temporal layers applying both convolutions and temporal attention along the time axis. This means the video clip length $T$ can be either fixed or vary during training. As we will introduce later in Section 3.3, it is a natural choice to use a fixed-length video clip for inference. This leads to an intuitive assumption that training under the same, fixed length $T$ leads to better performance. Surprisingly, our experimental results demonstrate that using a uniformly sampled clip length from $[1,T_{\text{max}}]$ yields better performance. We use this random clip-length sampling strategy by default.

Joint v.s. Sequential Spatial-Temporal Fine-Tune. The original SVD fine-tunes the full UNet $D_{\theta}$ on video datasets, meaning that both the spatial layers and the temporal layers are trained jointly. We investigate another training protocol – sequential spatial-temporal training. Specifically, we first train the spatial layers using single-frame supervision. After convergence, we keep the spatial layers frozen and fine-tune the temporal layers using clips of randomly sampled length as supervision. Surprisingly, this sequential training of spatial and temporal layers allows for achieving good spatial accuracy while improving temporal consistency, yielding better performance than training the full network.

Figure 2: Inference pipeline. We explore two inference strategies. Option 1: Separate Inference involves dividing the videos into non-overlapping clips and predicting their depths individually. Option 2: Temporal Inpaint Inference denotes inpainting later frames $\mathbf{z}^{(0pt_{[W:T]})}$ in a clip based on the previous frames’ prediction $\mathbf{z}^{(0pt_{[0:W]})}$. Our proposed temporal inpaint enhances temporal consistency.

3.3 Inference Strategy

Having trained our model using the aforementioned strategy, we are capable of generating depth videos with a high degree of consistency in a single inference. However, due to resource constraints, it is not possible to infer a long-horizon video in a single forward pass. With a single 48G A6000 and a resolution of $576\times 768$, the maximum number of frames we can process is $38$. As such, we have to segment the long video into several clips and proceed to infer each separately. However, when no information is shared across clips, the model may still suffer from inconsistency prediction across different clips. Therefore, it becomes highly important to minimize the gap between each inference result. We evaluate two different inference protocols to determine the best practice.

Separate Inference. Given a video sequence with a total frame number $T_{\text{seq}}$, we set the clip frame number to $T$ and cut the video into $K=\left\lceil T_{\text{seq}}/T\right\rceil$ clips (the number of frames in the last clip is padded to make it up to $T$). We then carry out $K$ times of inferences, subsequently concatenating all of these results. Conversely to Marigold [36], we do not run ensemble inference, as we found it ineffective when combined with EDM.

Temporal Inpaint Inference. To exchange temporal information across clips, we introduce a novel temporal inpainting inference strategy, inspired by spatial inpainting methods [46]. We add $W$ frames overlap between adjacent $T$-frame clips ($W<T$). For the first clip, we denoise from a randomly-sampled Gaussian noise using Eq. 4. Then, for the remaining clips, we retain the last $W$ depth maps predicted from the previous inference, i.e., those overlapped with the current clip, and include them as input to the current inference. Specifically, we add Gaussian noise on these $W$ depth latents at any step $i$, to obtain $\mathbf{z}^{(0pt_{[0:W]})}_{i}$ using Eq. 1 for the first $W$ frames of the current clip. We then use Eq. 4 for the remaining $(T-W)$ frames $\mathbf{z}^{(0pt_{[W:T]})}_{i}$. We get the final $\mathbf{z}^{(0pt)}_{i}$ via concatenating these two items along the time axis. Thus we achieve the following expression for one reverse step in our approach

$$\mathbf{z}^{(0pt_{[0:W]})}_{i}=\mathbf{z}^{(0pt_{[0:W]})}+\mathbf{n}_{i},\mathbf{n}_{i}\sim\mathcal{N}(0,\sigma^{2}_{i}I)$$ (5)

$$\mathbf{z}^{(0pt_{[W:T]})}_{i}=\left[\mathbf{z}^{(0pt)}_{i+1}+\frac{\mathbf{z}_{i+1}-D_{\mathbf{\theta}}(\mathbf{z}^{(0pt)}_{i+1};\sigma_{i+1},\mathbf{z}^{(\mathbf{x})})}{\sigma_{i+1}}(\sigma_{i}-\sigma_{i+1})\right]^{[W:T]},\ i<N$$ (6)

$$\mathbf{z}^{(0pt)}_{i}=concat(\mathbf{z}^{(0pt_{[0:W]})}_{i},\mathbf{z}^{(0pt_{[W:T]})}_{i}).$$ (7)

Our inference strategy is depicted in Figure 2. Intuitively, there is an efficiency-consistency trade-off for this inpainting strategy: a larger overlap $W$ yields more consistent performance but slower inference. We observe that a single-frame overlap ($W=1$) is able to improve consistency over the separate inference without much additional computation burden.

4 Experiment

4.1 Implementation Details and Datasets

Implementation Details. We implement ChronoDepth using diffusers and we fine-tune our video foundation model SVD, specifically the image-to-video variant, following the strategy discussed in Sec. 3.2. We disable the cross-attention conditioning of the original SVD. We use the standard EDM noise schedule and the network preconditioning [34]. We use images and video resolution of $576\times 768$. We first pre-train our model with single-frame images input for 20k steps using batch size 8 then fine-tune our model with five-frame video clips for 18k steps using batch size 1. The maximum video clip length $T_{\text{max}}$ is 5 for training. For the temporal inpainting inference, we set the length of each clip $T$ as 10 and the number of overlapping frames between two adjacent clips $W$ as 1. The entire training takes about 1.5 days on a cluster of 8 Nvidia Tesla A100-80GB GPUs. We use the Adam optimizer with a learning rate of 3e-5.

Training Datasets. We utilize four synthetic datasets, including single-frame and multi-frame datasets. 1) Single-frame: Hypersim [56] is a photorealistic synthetic dataset with 461 indoor scenes. We use the official split with around 54K samples from 365 scenes for training. We filter out incomplete samples and get 39K samples finally. 2) Multi-frames: Tartanair [67] is a dataset collected in simulation environments for robot navigation tasks including 18 scenes. We select 4 indoor scenes with 40K samples in total. Virtual KITTI 2 [11] is a synthetic urban dataset providing 5 scenes with various weather or modified camera configurations. We use 4 scenes with 20K samples for training. MVS-Synth [31] is a synthetic urban dataset captured in the video game GTA. We use all the 120 sequences with a total of 12K samples for training.

4.2 Evaluation

Evaluation Datasets. We perform a range of experiments to assess the performance of our model. We select several zero-shot video clips from KITTI-360 [44], MatrixCity [42] and ScanNet++ [74], with a total number of frames ranging from 200 to 240, instrumental for a thorough evaluation aimed at evaluating both spatial accuracy and temporal consistency in video depth estimation.

Metrics. We consider metrics for both spatial accuracy and temporal consistency. For spatial accuracy, we apply two widely recognized metrics [55] for spatial quality for depth estimation, Absolute Mean Relative Error (AbsRel) and $\delta 1$ accuracy with a specified threshold value of 1.25. For temporal consistency, we introduce multi-frame similarity (Sim.). Consider two depth maps $D^{m},D^{n}\in\mathcal{R}^{W\times H}$ at frame $m$, $n$ of the video sequence, we unproject $D^{m}$ into a point cloud. Next, using the ground-truth world-to-camera poses $P_{m},P_{n}\in\mathcal{R}^{3\times 4}$ for frames $m,n$’ camera, we can transform the point cloud from frame $m$’ camera space to frame $n$’ camera space, and project it onto frame $n$’s image plane to yield $D^{m\to n}$. We measure the temporal consistency as the average L1 distance between $D^{m\to n}$ and $D^{n}$. In practice, we only calculate multi-frame similarity on adjacent frames. Note, the ground-truth camera poses are for evaluation only.

4.3 Comparison Experiments

Baselines. We compare ChronoDepth with four baselines covering both monocular and video depth estimation, aiming at demonstrating its effectiveness across spatial accuracy and temporal consistency. For monocular depth estimation, we select baselines from state-of-the-art methods, which have been demonstrated generalizability across various benchmarks. These methods achieve satisfying spatial accuracy (DPT [54], DepthAnything [73], and Marigold [36]). For video depth estimation, we choose NVDS [69] which is a state-of-the-art video depth estimator based on DPT.

Quantitative and Qualitative Analisis. In Table 1, we compare ChronoDepth with both discriminative (top) and generative (bottom) existing methods. It is worth mentioning discriminative models are trained on millions of images annotated with ground truth depth, about one order of magnitude more than generative ones. Nonetheless, ChronoDepth consistently outperforms prior art in temporal consistency, while retaining comparable spatial accuracy, often higher than the Marigold baseline – conversely to our main video counterpart, NVDS. It’s notable that Marigold employs an ensemble strategy for optimal performance, which involves denoising iterations ten times over and subsequently averaging the predictions. However, this method significantly hampers the speed, stretching the inference time to 8.75 seconds per frame. In contrast, ChronoDepth’s inference process is markedly more efficient, demanding just 0.84 seconds per frame. Fig. 3 shows a qualitative comparison between the baselines and ChronoDepth on the three datasets involved in our evaluation. We can appreciate how our video depth estimation framework predicts sharper, finer, and more accurate details – as highlighted by the arrows. Further, we observe that ChronoDepth improves performance at far regions in driving scenarios thanks to the multi-frame context. Note that this is not reflected in the quantitative results due to the incomplete GT.

Method	KITTI-360				MatrixCity				ScanNet++
	AbsRel$\downarrow$	$\delta$1$\uparrow$	Sim.$\downarrow$		AbsRel$\downarrow$	$\delta$1$\uparrow$	Sim.$\downarrow$		AbsRel$\downarrow$	$\delta$1$\uparrow$	Sim.$\downarrow$
DPT [54]	17.7	75.4	2.25		11.9	86.2	10.5		13.9	81.8	0.17
NVDS [69]	31.7	62.1	2.92		13.4	85.8	11.3		17.3	72.0	0.14
DepthAnyting [73]	16.8	77.7	1.89		8.3	90.7	11.7		8.2	92.7	0.21
Marigold [36]	19.7	73.1	1.12		13.9	79.7	9.0		11.3	86.9	0.16
Marigold (w/o ensemble)	21.4	69.0	1.83		14.6	76.8	9.3		12.0	85.1	0.18
Ours	19.6	72.4	0.91		13.4	82.3	8.8		12.6	87.1	0.10

Table 1: Quantitative Comparison on zero-shot depth benchmarks. Top: Discriminative methods. Bottom: Generative methods.

Input RGB

NVDS [69]

DepthAnything [73]

Marigold [36]

Ours

Depth GT

Figure 3: Qualitative Comparison on KITTI-360 (rows 1-2), ScanNet++ (3-4) and MatrixCity (5-6). The first column displays RGB images, followed by depth maps predicted by NVDS, Depth Anything, Marigold, and ChronoDepth. Ground-truth depth is shown on the right-most column.

4.4 Ablation Studies

We then conduct ablation studies on KITTI-360 dataset.

	AbsRel$\downarrow$	$\delta$1$\uparrow$	Sim.$\downarrow$
w/o ImgDepth	24.2	59.8	0.94
w/o RandomClip	20.7	69.3	0.93
w/o S-T Fintune	21.0	68.7	0.92
w/o Inpainting	20.2	71.0	1.03
Full	19.6	72.4	0.91

Table 2: Ablation Study. We report accuracy and consistency metrics of our method on KITTI-360 without single frame image pretraining (w/o ImgDepth), without random clip length (w/o RandomClip), without Sequential Spatial-Temporal Fine-Tuning (w/o S-T fintune) and without inpainting during inference (w/o Inpainting)

Figure 4: Ablation Study. We report accuracy and consistency metrics of our method on KITTI-360 with different overlaps between clips. Overlap 0 refers to separate inference.

Single-Frame Datasets. We investigate the impact of single-frame data during the training phase. We train our model in two different settings, one with only multi-frame data and one with single-frame data additionally. As shown in Table 2, training with single-frame data significantly improves the video depth prediction in both spatial accuracy and temporal consistency over using merely multi-frame data (w/o ImgDepth). Specifically, training with single-frame data additionally yields better estimation in regions characterized by a high frequency of depth variations.

Random Clip Length Sampling. Next, we ablate the effectiveness of random video clip length sampling during training. Table 2 shows that removing this sampling (w/o RandomClip) leads to performance degradation in spatial accuracy. This indicates that the random clip length sampling strategy acts as an effective form of data augmentation, mitigating the risk of model overfitting.

Sequential Spatial-Temporal Fine-Tune. We evaluate the effect of sequential spatial-temporal training protocol in Table 2 by jointly training the full network (w/o S-T Finetune). The sequential training strategy leads to better spatial accuracy and temporal coherence, which means disentangling spatial and temporal layers could be the better way to tame video foundation into depth estimator.

Temporal Inpaint Inference. We compare the results of temporal inpaint inference and separate inference derived from the same checkpoint. For temporal inpaint inference, we set the overlap between adjacent clips to $1$. As shown in Table 2, this latter achieves significantly improved performance in temporal consistency with only an inconsequential compromise to spatial accuracy. Fig. 4 illustrates the trade-off between the performance of our method and the efficiency of inference, specifically regarding the number of overlapping frames during inpainting.

4.5 Applications

Depth Conditioned Video Generation. We perform a depth-conditioned video generation experiment. Video depth maps generated by our and other approaches served as inputs to a video ControlNet [81]. As shown in Fig. 5, we manage to synthesize high-quality RGB videos (depth maps generated on the KITTI-360 validation set, with the corresponding first frame RGB used as the reference image). Furthermore, we quantitatively assess the visual fidelity (FID [27]) and temporal consistency (FVD [63]) of depth map videos produced by various depth estimation methods as reported in Table 3.

Method	KITTI-360
	FID$\downarrow$	FVD$\downarrow$
DPT [54]	79.7	310.3
NVDS [69]	87.2	420.5
DepthAnyting [73]	59.0	322.3
Marigold [36]	122.5	683.5
Ours	63.1	292.4

Table 3: Quality of Video Generation using different depth estimators. Our temporally consistent depth yields videos of superior FVD compared to baselines.

Marigold [36]	Ours

Figure 5: Depth Conditioned Video Generation. Compared to baseline methods such as Marigold, the depth maps generated by method can drive ControlNet to produce more consistent and releastic videos.

Novel View Synthesis. We then utilize unprojected point clouds generated by ChronoDepth’s depth maps to initialize 3DGS [37]. As demonstrated in Fig. 6, ChronoDepth converges at a notably faster rate than our baseline. For a graphic evaluation, we exhibit a qualitative comparison in Fig. 7. The application of ChronoDepth as the foundation for NVS results delivers a superior reconstruction quality, particularly in distant regions.

Figure 6: PSNR Convergence Curve. We report PSNR of 3DGS over training iterations using different initializations.

Marigold [36]
Ours

Figure 7: Novel View Synthesis Comparison using different intializations.

5 Conclusion

This paper introduces ChronoDepth, a video depth estimator that prioritizes temporal consistency by leveraging video generation priors. Our exploration of various training protocols and evaluation methodologies has led us to identify the most effective approach, resulting in superior performance. Specifically, ChronoDepth outperforms existing methods in terms of temporal consistency, surpassing both image and video depth estimation techniques, while maintaining comparable spatial accuracy. Our assessments on downstream applications, namely depth-conditioned video generation and novel view synthesis, underscore the advantages of our temporally consistent depth estimation approach. We contend that our empirical insights into harnessing video generation models for depth estimation lay the groundwork for future investigations in this domain.