Dynamic LiDAR Re-simulation using Compositional Neural Fields

The static background is encoded into a neural field $\mathbf{F}_{\text{static}}:(\mathbf{x},\mathbf{d})\mapsto(s,e,p_{d})$ that estimates the signed distance $s$, intensity $e$, and ray drop probability $p_{d}\in[0,1]$ given the point coordinates $\mathbf{x}$ and the ray direction $\mathbf{d}$. In practice, we first use a multi-resolution hash encoding (MRH) [29] to map each point to its positional feature $\mathbf{f}_{\text{pos}}\in^{32}$, and project the view direction onto the first 16 coefficients of the spherical harmonics basis, resulting in $\mathbf{f}_{\text{dir}}$. Subsequently, we utilize three Multilayer Perceptrons (MLPs) to estimate the scene properties as follows:

Here, $f_{s},f_{e},$ and $f_{\text{drop}}$ are three MLPs, $\mathbf{f}_{\text{ray}}\in\mathbb{R}^{31}$ represents the ray feature and is constructed by concatenating the per-point geometric feature and the directional feature. The geometric feature is denoted as $\mathbf{f}_{\text{geo}}\in\mathbb{R}^{15}$. For implementation details please refer to the supplementary material.

LiDAR scans collected over time are often misaligned due to the motion of both the sensor and other objects in the scene. Despite applying ego-motion for aligning static background points, dynamic objects remain blurred along their trajectories. Our approach to constructing a dynamic neural scene representation is grounded in the assumption that each dynamic object only undergoes rigid motion. Therefore, we can first align them over time and reconstruct them in their canonical coordinate frame, and then render them over time by reversing the alignment of the neural field.

Specifically, consider a dynamic vehicle $v$ occurring in LiDAR scans $\{\mathbf{X}^{v}_{t}\}_{t=1}^{T}$ along with the associated bounding boxes $\{\mathbf{B}^{v}_{t}\in\mathbb{R}^{3\times 8}\}_{t=1}^{T}$ in the world coordinate frame. Here each bounding box is defined by its eight corners, and the first bounding box $\mathbf{B}^{v}_{1}$ is considered as the canonical box. We estimate the relative transformations $\{\mathbf{T}_{t}\in\text{SE}(3)\}_{t=2}^{T}$ between the remaining $T-1$ bounding boxes and the canonical one, expressed as $\mathbf{B}_{1}^{v}=\mathbf{T}_{t}\mathbf{B}_{t}^{v}$.^***$\mathbf{T}\mathbf{B}=\mathbf{R}\mathbf{B}+\mathbf{t}$, with $\mathbf{R}$, $\mathbf{t}$ the rotation/translation components of $\mathbf{T}$.. Subsequently, all LiDAR measurements on the object are transformed and accumulated in its canonical coordinate frame. The vehicle $v$ is then reconstructed in its canonical space, akin to the static background, using a neural field $\mathbf{F}^{v}$. To render the dynamic vehicle at timestamp $t$, the corresponding rigid transformation is applied to the queried rays. The dynamic neural field can thus be expressed as: $\mathbf{F}^{v}_{t}:(\mathbf{T}_{t}\mathbf{x},\mathbf{T}_{t}\mathbf{d})\mapsto(s,e,p_{d})$. The rendering process for $\mathbf{F}^{v}$ is the same as for the static neural field $\mathbf{F}_{\text{static}}$.

As outlined in Sec. 3.1, each dynamic neural field is reconstructed in its unique canonical space, defined by a corresponding canonical box. To determine neural fields that intersect a given ray at inference time, we begin by estimating the transformations $\{\mathbf{T}_{t}^{v}\}_{v=1}^{N}$, which convert coordinates from the world framework to each vehicle’s canonical space at timestamp $t$. These transformations are determined by interpolating the training set transformations using spherical linear interpolation (SLERP) [40]. Then, we apply the transformations to the queried ray and run intersection tests with the canonical boxes of the fields.

After identifying the $k+1$ neural fields that potentially intersect with a ray, we run volume rendering on each field separately, yielding $k+1$ distinct sets of LiDAR measurements. Next, we evaluate the ray drop probabilities across these fields. A ray is deemed dropped if all neural fields indicate a drop probability $p_{d}>0.5$. For rays not classified as dropped, we sort the estimated ranges in ascending order and select the nearest one as our final range prediction. The corresponding intensity value at the closest range is extracted from the same neural field.

We construct the Waymo Dynamic dataset by selecting four representative scenes from Waymo Open [42], with multiple moving vehicles inside. These scenes are comprised of 50 consecutive frames. For evaluation purposes, every 5^th frame is held out for testing, the remaining 40 frames are used for training.

We also evaluate our method on four static scenes as introduced in [19]. There are two settings, Waymo Interp applies the same evaluation protocol as Waymo Dynamic, while Waymo NVS employs a dedicated closed-loop evaluation to validate the real novel view synthesis performance. Please refer to NFL [19] for more details about this setting.

TownClean and TownReal are synthetic static scenes introduced in NFL [19]. They consist of 50 simulated LiDAR scans of urban street environments, using idealised and diverging beams, respectively.

To evaluate the LiDAR range accuracy, we employ a suite of four metrics: mean absolute errors (MAE [cm]), median absolute errors (MedAE [cm]), Chamfer distance (CD [cm]) and MedAE for dynamic vehicles (MedAE Dyn [cm]). For intensity evaluation, We report root mean square error (RMSE). In addition to our primary evaluations, we assess the re-simulated LiDAR scans’ realism through two auxiliary tasks: object detection and semantic segmentation. For object detection, we calculate the detection agreement [27], both for all vehicles (Agg. [%]) and specifically for dynamic vehicles (Agg. Dyn. [%]). Regarding semantic segmentation, we report recall, precision, and intersection-over-union (IoU [%]). It is important to note that the predictions on the original LiDAR scans serve as our ground truth, against which we compare the results obtained from the re-simulated scans.

Regarding LiDAR simulation on static scenes, NFL [19] and LiDARsim [27] are the two closest baselines to compare to. Additionally, we include i-NGP [29], DS-NeRF [8], and URF [37] for comparison. As for simulation on dynamic scenes, we compare to LiDARsim [27] and UniSim [58].^***We re-implement both LiDARsim and UniSim, since they are not open-sourced. Please refer to the supplementary material for implementation details.

Quantitative comparisons with baseline methods are detailed in Tab. 1. DyNFL reconstructs more accurate range than LiDARsim [27] and UniSim [58]. The improvement is largely due to our SDF-based neural scene representation, which incorporates the physical aspects of LiDAR sensing. Additionally, our method employs a ray drop test when rendering multiple neural fields, leading to a more accurate reconstruction of dynamic vehicles, as evidenced in Fig. 2 and further supported by the errors shown in Fig. 3.

In addition to dynamic scenes, we evaluate DyNFL against baseline methods in static scenarios, Tab. 2 and Fig. 4. DyNFL excels in reconstructing crisp geometry. A key observation is its superior performance on planar regions (e.g., the ground shown in Fig. 4), especially when compared to NFL [19], which also uses a neural field for surface representation. This improvement is largely due to the enhanced surface regularization provided by SDF-based surface modeling.

We begin by assessing the efficacy of our SDF-based volume rendering for active sensors, results are shown in Tab. 3. When compared to our baseline that uses the SDF-based volume rendering for passive sensing, DyNFL demonstrates enhanced performance in both synthetic (TownClean) and real-world (Waymo Interp and Waymo Dynamic) datasets, indicating the importance of considering the physical LiDAR sensing process when addressing the inverse problem.

To validate the efficacy of our two-stage neural field composition approach, we compare it with an alternative approach utilized in UniSim [58]. The results are shown in Tab. 1. UniSim [58] blends different neural fields by sampling points from all intersected neural fields, followed by a single evaluation of volume rendering to produce the final LiDAR scan. In contrast, our method independently renders from each intersecting neural field first, and then combines these measurements into a final output using a ray drop test (cf. Fig. 5). This approach leads to markedly improved geometry reconstruction, exemplified by our method halving the Median Absolute Error (MedAE) across all points. This enhancement is even more evident when focusing solely on points related to dynamic vehicles (cf. Fig. 3).

We examine the importance of constraining the SDF at surface points, cf. Sec. 5, with the Town Real and Waymo Interp datasets. The results shown in Tab. 4 suggest that our method yields improved geometry reconstruction by explicitly enforcing SDF values near zero at the LiDAR points.

We run the pre-trained FSDv2 [10] object detector on re-simulated LiDAR scans of the Waymo Dynamic dataset. Our results are compared against those from LiDARsim [27], with the findings detailed in Tab. 5 and Fig. 6. In summary, detections in scans generated with DyNFL exhibit better agreement with those in real LiDAR scans. In Fig. 10 we additionally show detection results in the noisy background. The detections in the synthetic scans almost perfectly match those in the original scans. This indicates a high fidelity of our re-simulations, with a low domain gap to actual scans.

We segment synthetic scans into semantic classes with the pre-trained SPVNAS model [45] and show results in Tab. 6. DyNFL improves over baseline methods according to most evaluation metrics, underscoring the realism of our re-simulated LiDAR scans.

Our explicit scene decomposition and flexible composition technique enable seamless insertion and removal of neural assets across scenes. As demonstrated in Fig. 8, we are able to replace a car from one scene with a truck from another scene, achieving accurate reconstruction of both geometry and intensity. In contrast, UniSim [58] struggles to preserve high-quality geometry. This highlights the potential of our approach to generate diverse and realistic LiDAR scans of autonomous driving scenarios.

DyNFL also facilitates the manipulation of moving objects’ trajectories, by simply adjusting their relative poses to the canonical bounding box. Representative results are shown in  Fig. 9. The high realism of our re-simulation is also indicated by the successful detection of inserted virtual objects.

For the Waymo Dynamic dataset, we take them from 4 scenes of Waymo Open Dataset [42]. There are multiple moving vehicles inside each scene. 50 consecutive frames are taken from each scene for our evaluation. The vehicles are deemed as dynamic if the speed is $>1\,$m/s. in any of the 50 frames. The corresponding scene IDs on Waymo Open Dataset for our selected scenes are shown as follows:

For the Waymo Dynamic NVS dataset, we use the same 4 scenes as chosen in Waymo Dynamic. We change the evaluation paradigm similar to Waymo NVS  [19] such that we first train the model on all 50 consecutive LiDAR frames then we synthesize 50 novel LiDAR frames with a sensor shift of 2 meters. We then train a new model on the new 50 synthetic LiDAR scans and evaluate against the original 50 LiDAR scans.

Our model is implemented based on nerfstudio[44]. For the static neural field, we sample $N_{s}=512$ points in total, with $N_{u}=256$ uniformly sampled points and $N_{i}=256$ weighted sampled points with 8 upsample steps. In each upsample step, 32 points are sampled based on the weight distribution of the previously sampled points. For each dynamic neural field, we sample $N_{s}=128$ points in total, with $N_{u}=64$ uniformly sampled points and $N_{i}=64$ weighted sampled points with 4 upsample steps. During training, we minimize the loss function using the Adam [21] optimiser, with an initial learning rate of 0.005. It linearly decays to 0.0005 towards the end of training. For the loss weights, we use $w_{\zeta}=3,w_{e}=50,w_{\text{drop}}=0.15,w_{s}=1$, and $w_{\text{eik}}=0.3$. The batch size is 4096 and we train the model for 60000 iterations on a single RTX3090 GPU with float32 precision.

We re-implement the LiDARsim [27] as one of our baselines. First, we estimated point-wise normal vectors by considering all points within a 20 cm radius ball within the training set. Following this, we applied voxel down-sampling [46], employing a 4 cm voxel size to reconstruct individual disk surfels at each point. The surfel orientation is defined based on the estimated normal vector. During inference, we apply the ray-surfel intersections test to determine the intersection points, thus the range and intensity values. We select a fixed surfel radius of 6 cm for the Waymo dataset and 12 cm for the Town dataset. To handle dynamic vehicles, we follow LiDARsim [27] by aggregating the LiDAR points for each vehicle from all the training frames and representing them in the canonical frame of each vehicle. During inference, we transform all the aggregated vehicle points from their canonical frames to the world frame and run ray-surfel intersection.

We re-implement UniSim’s [58] rendering process for LiDAR measurements by replacing our ray-drop test-based neural fields composition method with its joint rendering method. For every ray $\mathbf{r}(\mathbf{o},\mathbf{d})$, we begin by conducting an intersection test with all dynamic bounding boxes in the scene to identify the near and far limits. We then uniformly sample 512 points along each ray, assigning each point to either a dynamic neural field, if it falls within a dynamic bounding box, or to the static neural field otherwise. After sampling, we query the SDF and intensity values from the relevant neural fields. Finally, using the SDF-based volume rendering formula in Eq. 51 for active sensors, we calculate the weights and perform the rendering. Note that we use the same neural field architecture as in our method.

For a ray emitted from the origin $\mathbf{o}\in^{3}$ towards direction $\mathbf{d}\in^{3}$, the density $\sigma_{\zeta}$ at range $\zeta$ indicates the likelihood of light interacting with particles at that point $\mathbf{r}_{\zeta}=\mathbf{o}+\zeta\mathbf{d}$. This interaction can include absorption or scattering of light. In passive sensing, density $\sigma$ is a critical factor in determining how much light from the scene’s illumination is likely to reach the sensor after passing through the medium.

quantifies the likelihood of light traveling through a given portion of the medium without being scattered or absorbed. Density is closely tied to the transmittance function $\mathcal{T}(\zeta)$, which indicates the probability of a ray traveling over the interval $[0,\zeta)$ without hitting any particles. Then the probability $\mathcal{T}(\zeta{+}d\zeta)$ of not hitting a particle when taking a differential step $d\zeta$ is equal to $\mathcal{T}(\zeta)$, the likelihood of the ray reaching $\zeta$, times $(1-d\zeta\cdot\sigma(\zeta))$, the probability of not hitting anything during the step:

We solve the differential equation as follows:

Hence, for a ray segment between $\zeta_{0}$ and $\zeta$, transmittance is given by:

which leads to following factorization of the transmittance:

is the complement of transmittance and represents the fraction of light that is either absorbed or scattered in the medium. In a homogeneous medium with constant density $\sigma$ the opacity for a segment $[\zeta_{j},\zeta_{j+1}]$ of length $\Delta\zeta$ is given by $\alpha_{\zeta_{j}}=1-exp(-\sigma\cdot\Delta\zeta)$