Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty

Abstract

Robust and realistic rendering for large-scale road scenes is essential in autonomous driving simulation. Recently, 3D Gaussian Splatting (3D-GS) has made groundbreaking progress in neural rendering, but the general fidelity of large-scale road scene renderings is often limited by the input imagery, which usually has a narrow field of view and focuses mainly on the street-level local area. Intuitively, the data from the drone’s perspective can provide a complementary viewpoint for the data from the ground vehicle’s perspective, enhancing the completeness of scene reconstruction and rendering. However, training naively with aerial and ground images, which exhibit large view disparity, poses a significant convergence challenge for 3D-GS, and does not demonstrate remarkable improvements in performance on road views. In order to enhance the novel view synthesis of road views and to effectively use the aerial information, we design an uncertainty-aware training method that allows aerial images to assist in the synthesis of areas where ground images have poor learning outcomes instead of weighting all pixels equally in 3D-GS training like prior work did. We are the first to introduce the cross-view uncertainty to 3D-GS by matching the car-view ensemble-based rendering uncertainty to aerial images, weighting the contribution of each pixel to the training process. Additionally, to systematically quantify evaluation metrics, we assemble a high-quality synthesized dataset comprising both aerial and ground images for road scenes. Through comprehensive results, we show that: (1) Jointly training aerial and ground images helps improve representation ability of 3D-GS when test views are shifted and rotated, but performs poorly on held-out road view test. (2) Our method reduces the weakness of the joint training, and out-performs other baselines quantitatively on both held-out tests and scenes involving view shifting and rotation on our datasets. (3) Qualitatively, our method shows great improvements in the rendering of road scene details, as shown in Fig. 1. The code and data for this work will be released at https://github.com/SainingZhang/UC-GS.

1 Introduction

Autonomous driving simulation serves as a critical platform for scaling up to real-world deployment. Realistic rendering and novel view synthesis (NVS) for large-scale road scenes have become increasingly important in autonomous driving simulation, as they enable the synthesis of high-quality training and testing data at a significantly lower cost compared to using real-world data. This capability also benefits a wide range of applications, including digital cities [Li et al.(2024)Li, Fidler, Kanazawa, and Williams, Ye et al.(2024)Ye, Liu, Ye, Chen, Wang, Yan, Shi, Zhao, and Zhou], virtual reality [Xu et al.(2023)Xu, Agrawal, Laney, Garcia, Bansal, Kim, Rota Bulò, Porzi, Kontschieder, Božič, et al.], and embodied AI [Qiu et al.(2024)Qiu, Hu, Yang, Song, Fu, Ye, Mu, Yang, Atanasov, Scherer, and Wang].

Recently, NeRF [Mildenhall et al.(2021)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng, Ost et al.(2021)Ost, Mannan, Thuerey, Knodt, and Heide, Wu et al.(2023)Wu, Liu, Luo, Zhong, Chen, Xiao, Hou, Lou, Chen, Yang, et al., Turki et al.(2023)Turki, Zhang, Ferroni, and Ramanan, Yuan et al.(2024)Yuan, Mao, Yang, Liu, Wang, and Zhao, Wei et al.(2024)Wei, Wang, Lu, Xu, Liu, Zhao, Chen, and Wang] has greatly enhanced fidelity of NVS by parameterizing the 3D scene as implicit neural fields but suffers from slow rendering due to exhaustive per-pixel ray sampling process especially in large-scale road scenes. 3D Gaussian Splatting (3D-GS) [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis] achieves real-time rendering by rasterizing the learnable Gaussian primitives.

Figure 1: Qualitative results of our Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty and several baseline methods. The dataset is 1.6m test set of New York City. The quality improvement is highlighted by boxes.

However, the rendering quality for both NeRF and 3D-GS is highly dependent on the input views. On the contrary, current road scene datasets, such as KITTI [Geiger et al.(2013)Geiger, Lenz, Stiller, and Urtasun] or nuScenes [Caesar et al.(2020)Caesar, Bankiti, Lang, Vora, Liong, Xu, Krishnan, Pan, Baldan, and Beijbom], only contain car-view images, which are limited by the field of view and focus on the street-level local areas. The most related work may be MatrixCity [Li et al.(2023)Li, Jiang, Xu, Xiangli, Wang, Lin, and Dai], which offers both aerial and street-level city views, but the aerial imagery’s high altitude limits its ability to capture fine-grained road details. This mismatch in scale and granularity between global aerial views and local street views makes MatrixCity unsuitable for drone-assisted road scene synthesis.

To address the aforementioned issues, we dedicate to establish a new paradigm for drone-assisted road scene synthesis, aiming to overcome the limitations of input views by integrating an aerial perspective to provide a comprehensive global view of road scenes, in contrast to the localized perspective obtained from ground-level vehicle cameras. The aerial perspective can be captured using devices such as drones.

Since it is difficult to capture well-aligned view synthesis ground truth dataset for evaluation in real world, we first create a synthesized dataset (Fig. 2) comprising aerial-ground imagery with viewpoints that have similar levels of information granularity across large-scale road scenes. For the aerial perspective, we simulate drone flight trajectories and behavior patterns using AirSim [Shah et al.(2018)Shah, Dey, Lovett, and Kapoor]. For the ground-view images, we sample them to simulate the perspective and field of view from onboard cameras on vehicles.

Figure 2: General view of the synthesized dataset. 5°d means 5 degrees downward.

On the other hand, the training process of 3D-GS leads it to an imperfect modeling when it comes to significant different viewpoints, such as aerial and ground views. This limited generalization capability is exacerbated in our specific setting, making simultaneous fitting of aerial and ground data challenging. For instance, the inherent ambiguity in aerial-ground data and the information outside co-view regions dilute useful information. Naively incorporating aerial data into the joint training with ground data adversely affects the convergence of the Gaussian field and compromises rendering quality for NVS when shifting and rotating views in autonomous driving scenes.

To overcome this problem, we introduce a novel uncertainty-aware training paradigm that enables more effective use of aerial imagery and guides 3D-GS to focus on challenging areas where ground data alone may struggle. By excluding irrelevant portions in aerial perspectives, such as the upper floors of buildings, which are less related to street scenes, we successfully mitigate the ambiguity and improve the fidelity of NVS such as view shifting and rotation on street.

The uncertainty is first computed within the ground-view image space through an ensemble-based method, and then projected to the aerial space to assist the training of 3D-GS, which is named as cross-view uncertainty as a new concept. Extensive experiments demonstrate that our uncertainty-aware training paradigm for drone-assisted road scene synthesis out-performs the naive joint training with aerial and ground images, as well as training with only ground images, both quantitatively and qualitatively. Our method offers significant benefits for applications like autonomous driving simulation. To summarize, the contributions of our work include:

• We formalize the problem of drone-assisted road scene synthesis and craft a high-quality and appropriate dataset for this new and important problem;

• We propose an uncertainty-aware training strategy and are the first to demonstrate that cross-view uncertainty can facilitate a pixel-weighted training paradigm while prior works use all pixels of images equally for 3D Gaussians’ training;

• Through extensive experiments and evaluations, we demonstrate notably improved performance on both held-out tests and scenarios involving view shifting and rotation on road scene synthesis.

2 Related Works

2.1 3D Scene Representation

As the foundation for 3D computer vision, various 3D representations have been proposed to depict real-world scenes such as point cloud-based representation [Xu et al.(2022b)Xu, Cai, He, Lin, and Zhang, Mur-Artal and Tardós(2017)], voxel-based representation [Mao et al.(2021)Mao, Xue, Niu, Bai, Feng, Liang, Xu, and Xu, Yuan et al.(2022)Yuan, Xu, Liu, Hong, and Zhang], or implicit representation [Park et al.(2019)Park, Florence, Straub, Newcombe, and Lovegrove, Mildenhall et al.(2021)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng]. Among the implicit representation, NeRF [Mildenhall et al.(2021)Mildenhall, Srinivasan, Tancik, Barron, Ramamoorthi, and Ng] stands out as a groundbreaking neural rendering method that represents 3D scenes as continuous radiance fields parameterized by neural networks, taking coordinates and viewing directions as inputs. With the rise of NeRF, many efforts have been made to enhance its quality and efficiency [Yu et al.(2021)Yu, Ye, Tancik, and Kanazawa, Xu et al.(2022a)Xu, Jiang, Wang, Fan, Shi, and Wang, Deng et al.(2022)Deng, Liu, Zhu, and Ramanan, Barron et al.(2021)Barron, Mildenhall, Tancik, Hedman, Martin-Brualla, and Srinivasan, Barron et al.(2022)Barron, Mildenhall, Verbin, Srinivasan, and Hedman, Barron et al.(2023)Barron, Mildenhall, Verbin, Srinivasan, and Hedman, Liu et al.(2024)Liu, Hu, Yang, Chen, Wang, Chen, Cai, Gao, and Zhao, Yuan and Zhao(2024), Xu et al.(2024)Xu, Liu, Chen, Zhu, Yan, Shi, Zhao, and Zhou, Chen et al.(2023)Chen, Liu, Zhao, Zhou, and Zhang]. Recently, 3D Gaussian splatting (3D-GS) [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis] has been proposed as a novel 3D scene representation, utilizing a set of 3D positions, opacity, anisotropic covariance, and spherical harmonic (SH) coefficients to represent a 3D scene. Compared with NeRF, 3D-GS based methods [Yu et al.(2023)Yu, Chen, Huang, Sattler, and Geiger, Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai, Cheng et al.(2024)Cheng, Long, Yang, Yao, Yin, Ma, Wang, and Chen, Song et al.(2024)Song, Zheng, Yuan, Gao, Zhao, He, Gu, and Zhao, Yang et al.(2024)Yang, Zhu, Jiang, Ye, Chen, Zhang, Chen, Zhao, and Zhao], shows superior performance in rendering speed, fidelity, and training time. In this work, we also leverage 3D-GS as the scene representation to resolve the problem of drone-assisted road scene synthesis.

2.2 Uncertainty Modeling

Modeling uncertainty has been a long-standing problem in deep learning. Early works usually resolve uncertainty estimation through Bayesian Neural Network (BNN) [Neal(2012), Neapolitan et al.(2004)]. However, these methods can be computationally expensive and challenging to implement. Later, dropout-based methods [Gal and Ghahramani(2016), Milanés-Hermosilla et al.(2021)Milanés-Hermosilla, Trujillo Codorniú, López-Baracaldo, Sagaró-Zamora, Delisle-Rodriguez, Villarejo-Mayor, and Núñez-Álvarez, Tian et al.(2022)Tian, Luo, Zhao, and Zhou] have emerged as a computationally efficient alternative that adds dropout during inference to estimate uncertainty. Besides, ensemble-based methods [Lakshminarayanan et al.(2017)Lakshminarayanan, Pritzel, and Blundell, Liu et al.(2019)Liu, Paisley, Kioumourtzoglou, and Coull] have been proposed to model uncertainty by merging the prediction from multiple independently trained neural networks. As for the field of 3D scene representation, a series of works [Shen et al.(2021)Shen, Ruiz, Agudo, and Moreno-Noguer, Shen et al.(2022)Shen, Agudo, Moreno-Noguer, and Ruiz, Goli et al.(2023)Goli, Reading, Selllán, Jacobson, and Tagliasacchi, Shen et al.(2023)Shen, Ren, Ruiz, and Moreno-Noguer] have focused on quantifying uncertainty in the prediction of NeRF. For example, S-NeRF [Shen et al.(2021)Shen, Ruiz, Agudo, and Moreno-Noguer] employ a probabilistic model to learn a simple distribution over radiance fields, while CF-NeRF [Shen et al.(2022)Shen, Agudo, Moreno-Noguer, and Ruiz] learns a distribution over possible radiance fields with latent variable modeling and conditional normalizing flows. With the emergence of 3D-GS, SGS [Savant et al.(2024)Savant, Valsesia, and Magli] first addresses uncertainty modeling in 3D-GS, and integrates a variational inference-based method with the rendering pipeline of 3D-GS. CG-SLAM [Hu et al.(2024)Hu, Chen, Feng, Li, Yang, Bao, Zhang, and Cui] also introduce the uncertainty-aware 3D-GS to SLAM. In this work, we introduce a novel cross-view uncertainty training paradigm to facilitate the training of 3D Gaussians on road scenes.

3 Method

Fig. 3 depicts the overview of our method. In Sec. 3.1, we briefly introduce the basic principles of original 3D-GS. Next, we construct the first drone-assisted road scene dataset in Sec. 3.2. Then, Sec. 3.3 illustrates how we model cross-view uncertainty through an ensemble-based rendering paradigm and an uncertainty projection module. Finally, by incorporating the uncertainty map into the loss function, we can build an uncertainty-aware training module, which facilitates the training of 3D-GS (Sec. 3.4).

Figure 3: Overview of Drone-assisted Road Gaussian Splatting with Cross-view Uncertainty. We first adopt an ensemble-based rendering uncertainty to quantify the learning outcomes of 3D Gaussians on ground images. Next, the ground uncertainty is projected to the air to build the cross-view uncertainty. Subsequently, we introduce the cross-view uncertainty to the training of 3D Gaussians as weight for each pixel of aerial images in the loss function, together with the original rendering loss of 3D-GS for ground images.

3.1 Preliminaries

3D-GS [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis] represents a 3D scene by a set of differentiable 3D Gaussians, which could be efficiently rendered to images through tile-based rasterization.

Specifically, initialized by a bunch of Structure-from-Motion (SfM) points, each 3D Gaussian is defined as:

$$G(x)=e^{-\frac{1}{2}(x-\mu)^{T}\Sigma^{-1}(x-\mu)},$$ (1)

where $x\in\mathbb{R}^{3\times 1}$ is a random 3D position in the scene, $\mu\in\mathbb{R}^{3\times 1}$ stands for the mean vector of the 3D Gaussian, and $\Sigma\in\mathbb{R}^{3\times 3}$ refers to its covariance matrix. In order to maintain its positive semi-definite, $\Sigma$ is further formulated as $\Sigma=RSS^{T}R^{T}$, where $R\in\mathbb{R}^{3\times 3}$ is the rotation matrix and $S\in\mathbb{R}^{3\times 3}$ is the scaling matrix.

To render the Gaussians into the image space, each pixel $p$ is colored by $\alpha$-blending $N$ sorted Gaussians overlapping $p$ as:

$$c(p)=\sum_{i=1}^{N}c_{i}\alpha_{i}\prod_{j=1}^{i-1}\left(1-\alpha_{j}\right),$$ (2)

where $\alpha_{j}$ is calculated by multiplying the 2D Guassian projected from 3D Gaussian $G$ in $p$ with the opacity of $G$, and $c_{i}$ is the color of $G$. Through the differentiable tile-based rasterizer technique, all attributes of the Gaussians could be learnable and optimized end-to-end via training view reconstruction.

In this work, we utilize Scaffold-GS [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai] as our baseline, as it represents the SOTA among 3D-GS based methods in road scene synthesis tasks. However, we posit that our proposed strategy holds promise for application across other 3D-GS based methods as well.

3.2 Drone-assisted Road Scene Synthesized Dataset

Crucial for autonomous driving simulation, high-fidelity view synthesis for large road scenes is often hindered by poor road rendering due to reliance on limited car-view imagery. To address this, we introduce drone-assisted road scene synthesis, leveraging aerial images as an additional input for a better scene reconstruction.

We present a new benchmark for assessing aerial images in large-scale road scene synthesis, featuring both aerial and ground posed images. Using Unreal Engine, we create two high-fidelity scenes to simulate real-world road imagery. AirSim [Shah et al.(2018)Shah, Dey, Lovett, and Kapoor] controls drones and vehicles for precise trajectory generation, simulating real scenarios. With the trajectories of drones, we employ AirSim to simulate the camera perspectives and render corresponding image data through the Unreal Engine. For ground-view imagery, we utilize vehicle trajectories to generate forward-facing images. As shown in Fig. 2, to replicate real-world driving conditions, we capture front-view ground images at heights of 1.5m and 1.8m, while aerial-view images are collected at the height 20m with the angle of 60° downward from the front view (based on some tests). Additional test data at 1.6m and 1.9m heights evaluate perspective impact. Each scene includes a training set of 315 ground and 351 aerial images, and a test set of 36 ground images, aiming to simulate diverse driving scenarios for a more representative benchmark dataset.

(a)	(b)	(c)

Figure 4: Results for training with ground or aerial and ground images on various models. (G), (A+G) are training with ground or aerial and ground images.

3.3 Cross-view Uncertainty Modeling

Preliminary Experiments and Motivation. From the comparison between the dotted and solid lines of different colors in Fig. 4 (a/b/c), it is clearly that jointly training aerial and ground images mitigates the decline in metrics during road view shifting and rotation compared with merely training with ground images. However, aerial images do not enhance the result on the held-out test of road scene synthesis, as shown by the point in the held-out column of Fig. 4 (a/b/c). Weighting all pixels from aerial and ground images equally while training will let aerial images have same synthesis priority as road views do for 3D Gaussians. The areas that are non-overlapped with road scene and the areas that have little contribution to the road scene synthesis in the aerial images not only fail to enhance the effectiveness of road reconstruction but also pose more challenges to 3D Gaussians’ convergence. This leads to poor rendering quality in the ground perspective when jointly training aerial and ground images.

Baseline and Implementation. In order to enhance the rendering result of road views, we attempt to quantify the contribution of each pixel in the aerial image to the road scene synthesis. However, undertaking such a task is very challenging. We decide to approach from a different angle by quantifying the quality of the learning outcomes of ground images to infer the weight to each pixel in the aerial image during Gaussians’ training. To conveniently and plausibly compute the learning outcomes, we adopt an ensemble-based rendering uncertainty [Lakshminarayanan et al.(2017)Lakshminarayanan, Pritzel, and Blundell] paradigm to quantify the uncertainty of each pixel in the ground imagery. To be more specific, we train an ensemble of $M$ gaussian splatting(GS)s initialised from the structure from motion (SfM) generated by ground imagery. By interpreting the ensemble as a uniformly-weighted mixture model, the members’ predictions are combined through averaging, and the predictive uncertainty is expressed as the variance over the individual member predictions. With an ensemble of GSs, the expected color of pixel $p$ in a scene is

$$\mu_{\text{RGB}}(p)=\frac{1}{M}\sum_{k=1}^{M}c_{k}(p).$$ (3)

The predictive uncertainty can be expressed as the variance over the individual member predictions:

$$\sigma_{\text{RGB}}^{2}(p)=\frac{1}{M}\sum_{k=1}^{M}\left(\mu(p)-c_{k}(p)\right)^{2}.$$ (4)

$\mu_{\text{RGB}}$ and $\sigma_{\text{RGB}}^{2}$ can be calculated very easily by rendering the $M$ individual RGB images and calculating the mean and variance directly in pixel space. Both will be 3-vectors over the RGB colour channels.

We combine the variances from the colour channels into a single uncertainty value by:

$$u(p)=\frac{1}{3}\sum_{c\in\{RGB\}}\log(\sigma^{2}_{\text{RGB},(c)}(p)+1),$$ (5)

where $\sigma^{2}_{\text{RGB},(c)}(p)$ indicates the variance associated with colour channel $c$, $\log$ is the logarithmic transformation to smooth and tighten the values for further normalization process.

Cross-view Uncertainty Projection. To project the uncertainty map from ground-view to aerial-view, we test several methods. Neural field-based methods like NeRF and 3D-GS are prone to overfitting, so neural fields trained with ground uncertainty maps are unable to render high-quality uncertainty in the air. Besides, recently appeared end-to-end dense stereo model—DUSt3R [Wang et al.(2023)Wang, Leroy, Cabon, Chidlovskii, and Revaud] has set SoTAs on many 3D tasks, which could be used as a 2D-2D pixel matcher between aerial and ground images. In this way, uncertainty maps from ground are projected to air through matches between ground and aerial images, and by averaging the uncertainties at pixels with multiple matches, we build reasonable cross-view uncertainty maps for training. The visualization of the cross-view uncertainty map is shown in Fig. 5.

Figure 5: The first row shows the visualization of the cross-view uncertainty, and the second row shows the corresponding aerial data.

3.4 Uncertainty-aware Training

In this section, we elaborate on how we introduce the cross-view uncertainty map to the training process. $U_{k}(x)$ presents the uncertainty value on the pixel position $x$ of the $k$-th aerial image. First of all, we normalize all the uncertainty to the range (0, 1) like this:

$$U^{\prime}_{k}={(\frac{U_{k}}{max(U_{1}...U_{M})-min(U_{1}...U_{M})})}^{\frac{1}{n}},$$ (6)

where $n$ refers to a hyperparameter for taking the $n$-th root, with the purpose of enhancing the impact of non-zero values.

Then, we introduce the uncertainty map to the color and SSIM loss as a weight map:

$$\mathcal{L}_{\text{color}}=\frac{1}{HW}\sum_{x=1}^{HW}U^{\prime}(x)|\hat{C}(x)-C(x)|,$$ (7)

$$\mathcal{L}_{\text{SSIM}}=\text{mean}(U^{\prime}(1.0-\text{SSIM\_MAP}(\hat{C},C))),$$ (8)

where $\hat{C}(x)$ and $\hat{C}$ represent ground-truth color, $H$ and $W$ stand for height and width of the image, and SSIM_MAP is the structural similarity of the inference and the ground-truth. The final loss function is given by:

$$\mathcal{L}=(1.0-\lambda_{\text{SSIM}})\mathcal{L}_{\text{color}}+\lambda_{\text{SSIM}}\mathcal{L}_{\text{SSIM}}+\lambda_{\text{vol}}\mathcal{L}_{\text{vol}},$$ (9)

where $\mathcal{L}_{\text{vol}}$ is the volume regularization used in [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai] to encourages the neural Gaussians to be small with minimal overlapping.

The loss function achieves the cross-view uncertainty-aware training which weights the effects of each pixel of aerial images to better assist in the road scene synthesis.

4 Experiments

4.1 Experimental Setup

Dataset and Metrics. In order to ensure the authenticity of the simulation data, we use two realistic city scene model, hereafter as New York City (NYC) and San Francisco (SF), from Kyrylo Sibiriakov [Sibiriakov(2022)] and Tav Shande [Shande(2022)] to collect the data as mentioned in Sec. 3.2. In addition to the $960\times 480$ aerial and ground images, we also collected $1280\times 720$ (HD) aerial images for experiments. All models are trained on 1.5m and 1.8m ground images, respectively, and tested at the viewpoint of the front and 5° downward at 0.1 meter above the ground level. All results are measured by three metrics: PSNR, SSIM and LPIPS.

Baseline and Implementation. Through preliminary experiments among several methods, Scaffold-GS [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai] is selected as the baseline since its outstanding performance. All methods are trained for 900k iterations. Furthermore, we record the results of other SoTA methods in NVS like Nerfacto [Tancik et al.(2023)Tancik, Weber, Ng, Li, Yi, Wang, Kristoffersen, Austin, Salahi, Ahuja, et al.], 3D-GS [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis] and Mip-Splatting [Yu et al.(2023)Yu, Chen, Huang, Sattler, and Geiger].

For hyperparameters in eq. 9, $\lambda_{\text{SSIM}}=0.2$ and $\lambda_{\text{vol}}=0.001$ as in [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai]. For $n$ in eq. 6, the default value is $6$ and we will discuss it in Sec. 4.3.

4.2 Main Results

From preliminary experiments (Fig. 4), it could be easily concluded that when testing the view shifting and rotation, the metrics of all methods decline. The inclusion of aerial images helps to slow down this trend compared to training the ground data along, indicating that aerial images can provide more perspective-rich information to maintain the rendering ability of the neural field during the view shifting and rotation. However, in the held-out viewpoints testing, training with aerial images performs no better than merely training on ground images, thus failing to demonstrate the superiority of aerial images in terms of view shifting and rotation.

Tab. 1 reports comprehensive results of road view synthesis on our datasets. After the implementation of the cross-view uncertainty, the paradigm makes a great progress. The average growth of PSNR on the held-out test set is 0.68 (NYC), and 0.41 (SF) compared with SoTA methods training with ground images. The SSIM and LPIPS also make a significant improvement. When shifting views, the PSNR is about 0.90 (NYC) and 0.80 (SF) more than training with ground images. The SSIM and LPIPS exhibit similar advancement trends. All our results out-performs previous SOTA solutions and Fig. 1 shows the improvement of our methods on certain details of road scene. In a word, our method not only enhances the representation of GS from ground perspectives but also improves the quality of road scenes synthesis during the view shifting and rotation.

Test set	Held-out			View(+0.1m)			View(+0.1m 5°down)
Method/Metrics	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Nerfacto(G) [Tancik et al.(2023)Tancik, Weber, Ng, Li, Yi, Wang, Kristoffersen, Austin, Salahi, Ahuja, et al.]	23.54	0.719	0.245	20.19	0.663	0.258	19.69	0.632	0.300
3D-GS(G) [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis]	23.71	0.706	0.363	20.54	0.688	0.346	20.01	0.646	0.387
Mip-Splatting(G) [Yu et al.(2023)Yu, Chen, Huang, Sattler, and Geiger]	25.35	0.779	0.302	20.51	0.710	0.302	20.03	0.668	0.350
Scaffold-GS(G) [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai]	25.64	0.790	0.265	22.19	0.746	0.281	21.55	0.705	0.326
Scaffold-GS(A+G)	25.66	0.782	0.273	22.56	0.744	0.286	22.10	0.709	0.328
Scaffold-GS(A*+G)	25.68	0.784	0.274	22.91	0.751	0.284	22.38	0.715	0.326
Ours	26.32	0.802	0.244	23.11	0.766	0.258	22.49	0.725	0.303

(a)

Table 1: Results on NYC (a) and SF (b). A* is HD aerial images. (G), (A+G) are training with ground or aerial and ground images.

Test set	Held-out			View(+0.1m)			View(+0.1m 5°down)
Method/Metrics	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
Nerfacto(G) [Tancik et al.(2023)Tancik, Weber, Ng, Li, Yi, Wang, Kristoffersen, Austin, Salahi, Ahuja, et al.]	23.82	0.631	0.344	23.38	0.618	0.344	22.77	0.587	0.421
3D-GS(G) [Kerbl et al.(2023)Kerbl, Kopanas, Leimkühler, and Drettakis]	23.22	0.630	0.449	21.96	0.597	0.457	21.64	0.564	0.492
Mip-Splatting(G) [Yu et al.(2023)Yu, Chen, Huang, Sattler, and Geiger]	24.95	0.690	0.381	23.38	0.654	0.391	22.90	0.613	0.434
Scaffold-GS(G) [Lu et al.(2023)Lu, Yu, Xu, Xiangli, Wang, Lin, and Dai]	25.16	0.697	0.375	24.37	0.675	0.386	23.69	0.637	0.423
Scaffold-GS(A+G)	24.98	0.691	0.383	24.63	0.683	0.384	23.97	0.647	0.421
Scaffold-GS(A*+G)	24.99	0.689	0.386	24.70	0.684	0.385	23.88	0.649	0.422
Ours	25.57	0.723	0.337	25.18	0.715	0.338	24.55	0.678	0.376

(b)

4.3 Ablation studies

Efficacy of Cross-view Uncertainty. Compared with equally training all aerial and ground images Tab. 1, the cross-view uncertainty-aware training achieves a 0.66 (NYC) and 0.59 (SF) increase in PSNR on held-out test set, and about 0.47 (NYC) and 0.57 (SF) when the view shifting and rotation. Moreover, our method also reverses the adverse effects of the joint training on SSIM and LPIPS, resulting in improvements in both metrics. It is also very impressive that our method performs even better than using HD aerial data when the view shifting and rotation. This reflects the effective utilization of aerial data in road scene synthesis. From Fig. 6, it is clear that our method not only contributes to the rendering effect of road textures but also enhances the clarity of roadside obstacles, lane markings and ground signs, which will greatly aid in autonomous driving simulation.

Figure 6: Rendering results for the ablation study of cross-view uncertainty. NYC: 1.6m (row 1), 1.6m 5° down (row2); SF: 1.9m (row 3), 1.9m 5° down (row4). A* is HD aerial images. (G), (A+G) are training with ground or aerial and ground images.

Efficacy of Hyperparameter $n$. Tab. 2 presents the experimental results for different values of $n$ in eq. 6. When $n$ is set to 1 (i.e., no $n$), there is no significant improvement in the metrics compared to equally training on all aerial and ground images. However, as $n$ increases to 2 or greater, the metrics improve with the increment of $n$, and the results become stable when $n\geq 6$. This indicates that when $n\geq 6$, the potential of aerial imagery is fully realized.

Test set	Held-out			View(+0.1m)			View(+0.1m 5°down)
$n$/Metrics	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$	PSNR $\uparrow$	SSIM $\uparrow$	LPIPS $\downarrow$
1	25.64	0.748	0.310	23.71	0.723	0.320	23.11	0.685	0.360
2	25.72	0.754	0.301	23.83	0.729	0.309	23.24	0.690	0.353
3	25.80	0.758	0.298	23.93	0.733	0.305	23.39	0.694	0.347
4	25.85	0.759	0.296	24.04	0.735	0.303	23.47	0.698	0.344
6	25.94	0.762	0.291	24.14	0.741	0.298	23.52	0.701	0.339
8	25.89	0.761	0.291	24.13	0.739	0.298	23.47	0.700	0.339
10	25.91	0.763	0.290	24.20	0.742	0.296	23.55	0.704	0.337

Table 2: The results for the ablation study of hyperparameter $n$. $n$ is for taking the $n$-th root to the uncertainty map. Metrics are averaged over testing on two datasets.

5 Conclusion

In this work, we propose a novel drone assisted road Gaussian Splatting with cross-view uncertainty. To use the global information from images in drones’ view to assist ground-view training, we are the first to introduce the cross-view uncertainty into the 3D-GS based model for weighting pixels in aerial images during training. This method reduces the impact of superfluous aerial information and effectively utilizes aerial images for road scene synthesis. From the experimental results, we achieve SoTA on two high-fidelity synthesized datasets. Our method enhances various metrics for held-out ground view synthesis while maintaining the robustness of aerial-ground training during the view shifting and rotation. The superiority of the method shows a great potential for the improvement of autonomous driving simulation in the near future.