3D Data Augmentation for Driving Scenes on Camera

Abstract

Driving scenes are extremely diverse and complicated that it is impossible to collect all cases with human effort alone. While data augmentation is an effective technique to enrich the training data, existing methods for camera data in autonomous driving applications are confined to the 2D image plane, which may not optimally increase data diversity in 3D real-world scenarios. To this end, we propose a 3D data augmentation approach termed Drive-3DAug, aiming at augmenting the driving scenes on camera in the 3D space. We first utilize Neural Radiance Field (NeRF) to reconstruct the 3D models of background and foreground objects. Then, augmented driving scenes can be obtained by placing the 3D objects with adapted location and orientation at the pre-defined valid region of backgrounds. As such, the training database could be effectively scaled up. However, the 3D object modeling is constrained to the image quality and the limited viewpoints. To overcome these problems, we modify the original NeRF by introducing a geometric rectified loss and a symmetric-aware training strategy. We evaluate our method for the camera-only monocular 3D detection task on the Waymo and nuScences datasets. The proposed data augmentation approach contributes to a gain of $1.7\%$ and $1.4\%$ in terms of detection accuracy, on Waymo and nuScences respectively. Furthermore, the constructed 3D models serve as digital driving assets and could be recycled for different detectors or other 3D perception tasks.

Equal contribution.

Appendices

Appendix

A Experiments

1.1 A.1 Implementation Details

Building Digital Driving Asset. We use the SOLO v2 [30] trained on COCO as the instance segmentation model for scene decomposition. We consider vehicle (Waymo) or car (nuScenes) as the foreground objects, since they are the most important components in driving scenes For 3D model reconstruction, we use the same model configuration as the DVGO [26] with our proposed techniques. We use 30–40 consecutive frames spanning an area of about 100-200 meters as one background and train each background model with 40,000 iterations. For the background voxel grid, we set the resolution as $330^3$ with a voxel size of 0.25-0.3m. The object models are trained with 20-60 consecutive frames, and we set the voxel size as 0.25m consistent with the background voxel size. The grid point number is about 1,000. Considering the construction cost and limitation of NeRF model for extreme illumination conditions, we select a subset of 100 sunny scenarios for each dataset. We use them for the data augmentation.

Applying Data Augmentation. Our method generates new data through rendering the recomposed scenes of the randomly selected 3D models. We manipulate the object in 3D space by 3D location jittering and orientation jittering and place it on the valid region of arbitrary backgrounds. The location jittering is defined by the maximum translation $(T_x, T_y)$ along the x-direction and the y-direction, and the orientation jittering is given by maximum rotation angle $T_{\theta }$. We consider two data augmentation strategies to valid the effectiveness of adding rotation and translation for 3D perception. The first is ${{\boldsymbol{Drive-3DAug w/o RT}}}$, in which we set translation $T_x=0$, $T_y=0$, and rotation $T_{\theta }=0$ and 1-2 new objects are arbitrarily pasted into the background scene on average. The second is ${{\boldsymbol{Drive-3DAug w/ RT}}}$ with $T_x=20$m, $T_y=5$m, and $T_{\theta }=30^{\circ }$. As for determining the valid region in the background scene, we set the pillar $Z_p$ resolution as 2m$\times$2m, $\delta _1=30$, and $\delta _2=15$ in Eq. 6. We generate 12 new images for every background model. This progress is offline data augmentation and these images are repeatedly used by different detectors.

Training Detectors. Two typical camera-based monocular 3D object detectors, FCOS3D [29] and SMOKE [19] are utilized to investigate the performance of our proposed method as they are two of the most popular and commonly used mono3D detectors. We maintain the same hyperparameters of detectors for all experiments in our study, introduced in the supplementary. We use all the scenes in the training set to train the detectors and evaluate the detectors on the entire validation set. However, we sample training data every 3 frames on Waymo during training due to limited computational resources. Besides, although our method can be applied to any view of cameras, we only take into account images taken by the front camera on Waymo and nuScenes in this work because of the computational resource. For the usage of the generated images, we randomly replace the image in a batch with the augmented data if it belongs to the scenes in the digital driving asset.

1.2 A.2 Benchmark Details

Waymo Dataset. The Waymo Open Dataset [27] is a large-scale dataset for autonomous driving that contains 798 scenes in the training dataset and 202 scenes in the validation dataset. The image resolution for the front camera is $1920 \times 1280$. Waymo uses the LET-AP, the average precision with longitudinal error tolerance, to evaluate detection models. Besides, Waymo also adopts the LET-APL and LET-APH metrics, which are the longitudinal affinity weighted LET-AP and the heading accuracy weighted LET-AP, respectively.

nuScenes Dataset. The nuScenes [2] is a widely used benchmark for 3D object detection. It contains 700 training scenes and 150 validation scenes. The resolution of each image is 1600 $\times$ 900. As for the metrics, nuScenes computes mAP using the center distance on the ground plane to match the predicted boxe and the ground truth. It also contains different types of true positive metrics (TP metrics). We use ATE, ASE and AOE in this paper, for measuring the errors of translation, scale and orientation, respectively.

1.3 A.3 Ablative Studies

Depth Supervision. We qualitatively and quantitatively investigate the effect of depth supervision on background model training and 3D augmentation. Table 4 shows that the 3D augmentation based on background model trained with depth supervision has better performance, with LET-AP (0.590 vs. 0.585). Figure 8 shows that NeRF can reconstruct the background with high quality given depth supervision, and the 3D background model quality can be decreased without depth information. Thus, LET-AP, LET-APH and LET-APL on car have a slight decrease with 0.001 for 3D augmentation using background model without supervision.

Table 4. Ablation Study of Drive-3DAug for FCOS3D on Waymo validation set. 3DAug means we use DVGO [26] for data augmentation. DS means depth supervision.

Fig. 8.

Visualization of rendered depth map. The model with depth supervision depicts better performance.

Reconstruction Cost. Table 5 depicts the comparison of reconstruction speed on a NVIDIA V100 GPU between previous methods and our method. Unlike the MLP-based NeRF [20] which needs more than 20 hours of training for one background, it takes about 0.5h for the voxel-based NeRF with depth supervision. For the object model, the reconstruction time is within minutes. Considering the model size of our NeRF is rather small, we can run multiple reconstructions in parallel. Moreover, once these models are trained, they could be recycled for different detectors as a digital driving assets.

Table 5. Reconstruction cost of different methods for one background, assessed on a NVIDIA V100 GPU with 16GB memory.

B Discussion

Driving scenes are extremely complicated, including lots of object categories under different illumination and weather conditions. Currently, our method only augments limited classes of objects under good illumination conditions. It is worth to include more situations as the digital driving asset for the future work. Besides, our method only changes the object with geometric transformation. More augmentation strategies, such as manipulating the background or changing the appearance of objects are also worth for further exploration.

Fig. 9.

Corner Cases generated by the Drive-3DAug, where (a), (b) illustrate occlusion situations, (c) demonstrates strange vehicle heading direction, and (d) shows a vehicle appended on slope road. Cars with 3D bounding boxes are rendered by NeRF.

Corner Case Generation. Drive-3DAug is able to generate many photographic data for various corner cases without much effort for autonomous driving systems. As described in Fig. 9, we use our method to simulate several corner cases including the car occluded by the environment, the car appearing on the road with strange positions and headings, and the car on the slope, which are hard to collect in the real world. This shows that 3D data augmentation can help alleviate issues of autonomous driving caused by plenty of corner cases.