eGAC3D: enhancing depth adaptive convolution and depth estimation for monocular 3D object pose detection

Data-lifting approaches

This line of works researches how to transform the monocular image to mimic the 3D point cloud input of LiDAR-based detection methods. Common approaches usually generate pseudo point clouds from the pixel-wise depth estimation of RGB images and converting perspectives images to birds-eye-view (BEV) images. Then, a 3D object detector is applied to perform the detection task from the pseudo point clouds like LiDAR-based approaches. Representation transformation-based approaches significantly improve the performance of monocular object detection results. However, these approaches require many complex computing stages and take long inference time. Wang et al. (2019) proposed a method using state-of-the-art depth estimation to generate pseudo point cloud before directly applying LiDAR-based 3D object detection algorithms. By leveraging the estimated depth map D as an additional signal along with the RGB image, the 3D location (x, y, z) of each pixel (u, v) could be derived as follows:

(1) $$x={\displaystyle \frac{(u-c_U)z}{f_U}}$$

(2) $$y={\displaystyle \frac{(v-c_V)z}{f_V}}$$

(3) $$z=D(u,v)$$where ( $c_U,c_V$) is the pixel location corresponding to the camera center and $f_U,f_V$ are the vertical and horizontal focal length.

You et al. (2020) offered Pseudo-LiDAR++ as an enhancement to their previous work (Wang et al., 2019). They modified the stereo network architecture and loss function in order to make them more aligned with a precise depth estimation of distant objects. Then, they utilized highly sparse, but less expensive LiDAR sensors for debiasing depth estimation and improving the correctness of point cloud representation.

Wang et al. (2020) introduced a noteworthy method called ForeSeE based on pseudo-LiDAR approaches. According to the authors’ finding, foreground and background have different depth distributions so their depth may be separately determined by using distinct optimization objectives and decoders. This work significantly improves the depth estimation performance on foreground objects.

Feature-lifting approaches

Roddick, Kendall & Cipolla (2019) proposed the orthographic feature transform (OFT), which constructs an orthographic birds-eye-view from the corresponding monocular pictorial cues. Specifically, each voxel of the 3D feature map is the average pooling of the image-based features over the projected voxel area. Then, in order to preserve the computational efficiency, they accumulate the 3D voxel along the vertical axis to create the birds-eye-view orthographic features. Finally, the orthographic features are processed by the topdown convolutional network to estimate the objects’ locations and poses directly.

Recently, Reading et al. (2021) have introduced a new method following the representation transformation approach named CaDDN. The key idea is to project the perspective image to BEV representation and then perform object detection. In this work, rich contextual feature information is projected to the appropriate depth interval in 3D space based on a predicted categorical depth distribution for each pixel. However, instead of separating depth estimation from 3D detection during the training phase like previous methods, the authors performed end-to-end training to focus on generating an interpretable depth representation for monocular 3D object detection.

Result-lifting approaches

Result-lifting methods leverage the existing 2D object detection frameworks to directly estimate the object’s 2D location with its corresponding depth, dimension, and orientation from 2D features. Liu, Wu & T’oth (2020) employed the CenterNet (Zhou, Wang & Krähenbühl, 2019) architecture to predict object’s projected 3D center with its corresponding depth, dimension and orientation. Then, they reconstructed the object’s 3D location via 2D-3D projection. The authors also introduced a multi-step disentangling loss to enhance the optimization robustness. Zhou et al. (2021) improved the above design by incorporating object instance segmentation to collect the instance-aware 2D features extracted from the backbone. Instead of directly predicting the projected 3D keypoints from 2D features, Li et al. (2020) estimated the offset of the keypoints to the 2D object’s center. They formalized a post-optimization process from the object’s keypoints, dimensions, and orientation to retrieve the final object’s location in 3D space. To perform an end-to-end training pipeline, Li & Zhao (2021) transformed the post-optimization step to a differentiable loss and integrated it into the training phase. Ma et al. (2021) found that far-away objects are almost impossible to localize accurately by using only monocular image and adding the samples in the training phase make the model hard to be optimized. They proposed the hard and soft codings to handle improper samples, where the hard coding eliminates all the distant objects by a fixed threshold, and the soft coding re-weights the impacts of training objects according to their ground-truth distance.

Brazil & Liu (2019) exploited the anchor-based object detection approach for monocular 3D detection. The authors introduced a single-stage network with the concept of a 2D-3D anchor box to predict 2D and 3D boxes simultaneously. Furthermore, they introduced depth-aware convolution to improve the spatial awareness of high-level features. Ding et al. (2020) also followed the 3D anchor-based design and utilized the external depth maps guidance to learn the geometric-aware deep representation for the detection network. Besides, Liu, Yixuan & Liu (2021) proposed another strategy to minimize the gap between 2D and 3D representation by leveraging the fact that most objects of interest should be on the ground plane. The authors accumulated the 2D center’s feature with the corresponding contact point’s feature to gather the useful structural information. Kumar, Brazil & Liu (2021) designed a differentiable non-maximum-suppression (NMS) for monocular 3D object detection and directly calculate the loss on the results after NMS step.

Depth adaptive convolution with variant guidance

The depth adaptive convolution layer proposed in our previous work GAC3D (Bui et al., 2021) injects information from depth predictions using pixel-adaptive convolution from Su et al. (2019). We design the detection head with a sequence of a depth adaptive convolution layer with $3\times 3$ filters, a ReLU activation, and a $1\times 1$ standard convolution layer. The depth maps generated from pre-trained models are scaled down by the bilinear interpolation with the factor of ${\displaystyle \frac{1}{4}}$. The scaled depth estimations are then directly used as the guidance for depth adaptive convolution. The kernel’s parameters, such as padding values, strides remain unchanged. Hence, each depth adaptive detection head has the exact size of input and output.

On account of the 2D-3D projection properties, the monocular pixels of far distance objects would critically scale in terms of depth value. Hence, treating every pixel-wise depth with the exact guidance is not sufficient enough. We improve the depth adaptive convolution by introducing a learnable guiding kernel called variant guidance in eGAC3D. This innovative guidance allows the network to learn the scale sensitivity of monocular depth estimation implicitly and smoothly adapt the guidance weights basing on both the local object features and global structure information. Figure 5 illustrates the proposed Depth Adaptive Convolution with Variant Guidance in eGAC3D while its formulation is adjusted by (4).

(4) $$\{\begin{array}{ll}{\mathbf{d}}^{\mathbf{\ast }}& =G(d)\\ {{\mathbf{v}}_{\mathbf{i}}}^{\mathrm{\prime }}& ={\sum }_{j\in \mathrm{\Omega }(i)}K({\mathbf{d}}_{\mathbf{i}}^{\ast },{\mathbf{d}}_{\mathbf{j}}^{\ast })\mathbf{W}[{\mathbf{p}}_{\mathbf{i}}-{\mathbf{p}}_{\mathbf{j}}]{\mathbf{v}}_{\mathbf{j}}+\mathbf{b}\end{array}$$where G is the variant guidance function to compute a variant vector ${\mathbf{d}}^{\mathbf{\ast }}$ of the depth estimation value d. We implement G by applying a sequence of standard convolution layers and ReLU activations. G comprises two consecutive block of $3\times 3$ convolution with the stride of two followed by ReLU activation and a $1\times 1$ convolution layer.

Depth self-estimation

The investigation by Wang et al. (2020) showed that foreground depth and background depth of traffic scenes show different data distributions. As our foremost task is 3D object detection, the precision of the foreground depth plays a crucial role in object localization. However, pre-trained depth estimators typically penalize errors across all pixels equally. Hence, using pre-trained depth estimators would lead to over-emphasizing non-object pixels problems and consequently decrease the object detection performance. Moreover, employing a separate depth estimator requires redundant computation cost and critically increases the framework’s inference time. To overcome the above issues, we design an end-to-end training strategy for depth guided detection network. In eGAC3D, we create a new head in the detection network to estimate depth values itself as shown in Fig. 3 instead of leveraging an external depth estimator trained on a generic depth estimation image corpus as our previous work GAC3D.

The depth branch processes the feature maps with a $3\times 3$ convolution with ReLU activation followed a $1\times 1$ convolution with Sigmoid activation to produce a low resolution of depth estimation $\in R^{{\displaystyle \frac{H}{4}\times {\displaystyle \frac{W}{4}\times 1}}}$. Then, it is interpolated with the nearest neighbor mode to produce the final depth estimation with full resolution $\in H\times W\times 1$.

Geometric ground-guide module

In eGAC3D, we adopt this Geometric Ground-Guide Module (GGGM) proposed in GAC3D to leverage the output from the detection network, including orientation, dimension, keypoints’ position, and pseudo-contact point’s position for reconstructing the 3D bounding box. GGGM begins by estimating a pseudo-position for each detected object. The detection network outputs including the 2D center, dimension, orientation, nine predicted keypoints, and the pseudo-position are then used to generate a system of geometric equations to compute the position of the 3D bounding box center. Figure 6 depicts an example of how GGGM works, while the details of its computation were presented in GAC3D.

Confidence score

The confidence score directly affects the evaluation of detection results. In the 2D object detection scope, the confidence score denotes the probability that a bounding box containing an object. A high-confidence object also indicates that it could be easily discriminated from the 2D image. However, because of the occlusion, truncation, and distance variance of objects in 3D space, objects assigned with a high 2D confidence score could not be located easily in 3D space. Thus, it is inadequate to naively employ original 2D object detection confidence as a 3D confidence score. Moreover, due to the lack of label information, we could not train our model to directly regress the confidence of a 3D detection. To resolve the current problem, we follow the object confidence decomposition proposed in Li & Zhao (2021). This 3D object confidence score estimation combines both 2D and 3D perspective detection mechanism by computing a prior distribution as follows:

(5) $$conf=con{f}_{2D}\times con{f}_{3D}$$where con f is the object’s confidence score, $con{f}_{2D}$ is the 2D confidence score and extracted from the output heatmap of center head. The 3D confidence head predicts the 3D bounding box confidence $Con{f}_{3D}\in R^{{\displaystyle \frac{H}{4}\times {\displaystyle \frac{W}{4}\times 1}}}$ by the IoU between estimation and groundtruth. Figure 7 gives an example of the correlation between 3D confidence scores estimated from our detection network and the quality of the predicted bounding boxes.

Implementation

Similar to our previous work GAC3D, we implement and train eGAC3D on a machine equipped with an Intel Core i7-9700K CPU, a single RTX-2080ti GPU 12 GB, and 32 GB main memory. For DLA-34 backbone network, we first load the ImageNet pre-trained weights which are provided in the Pytorch framework by default. The other weights of our framework are trained from scratch. Our DLA-34 backbone networks contain 16.85 million parameters.

Evaluation

We conduct experiments to evaluate our eGAC3D performance on the official KITTI object detection benchmark (Geiger, Lenz & Urtasun, 2012) that consists of four different benchmarks, including 2D detection, orientation similarity, 3D detection, and BEV detection on three classes (Car, Pedestrian, and Cyclist). These benchmarks are separated into three difficulty levels Easy, Moderate, and Hard according to the bounding box’s height, occlusion and truncation level (Geiger, Lenz & Urtasun, 2012). All benchmarks are ranked by the precision-recall curves given the IoU threshold or rotation threshold. From October 2019, following Chen et al. (2015), the evaluation metrics are changed from 11-point Interpolated Average Precision (AP) metric AP ${|}_{R11}$ to 40 recall positions-based metric AP ${|}_{R40}$. For evaluating 3D detection performance, we mainly focus on the 3D detection, and BEV detection metrics, especially for the Car category. For Car objects, it requires an 3D bounding box overlap of 70% and for Cyclist and Pedestrian is 50%. Table 1 shows the detailed evaluation results of eGAC3D performance. We also highlight the 3D detection and BEV detection on the Car category that is the main consideration in our study.

Deployment process

Figure 8 illustrates the deployment cycle of our eGAC3D on an embedded platform, Jetson Xavier NX. As above presented, the model was trained and tested on a mainframe computer with GPU supported. When the model was properly tested and achieve appropriate results, we bring the model weight to the Jetson board by using ONNX parser to convert model’s weights from Pytorch format (.pth) to ONNX format (.onnx).

With TensorRT, models can be imported from frameworks like TensorFlow and PyTorch via the ONNX format (.onnx) to the Jetson platform. They may also be created from scratch by instantiating individual layers and setting parameters and weights directly. In the proposed eGAC3D, we adopt a Deformable Convolution Network (DCN) in Detection Heads that is not natively supported in TensorRT. Therefore, we need to perform the model optimization to convert the model to TensorRT before building the TensorRT Engine. Since TensorRT already provides the interface class IPluginV2Ext for users to extend and implement the custom layer, we create a network plugin to implement our custom layer and add it to the engine in build time.

Regarding data calibration, TensorRT supports three types of parameter precision including 32-bit floating point (FP32), 16-bit floating point (FP16), and 8-bit integer (INT8) as described in Table 6. During the training process, parameters and activation are represented in FP32 because FP32’s high precision fares well for every training step that usually corrects the parameters by a small amount. However, in the inference phase, 16-bit or 8-bit representation of parameters and activation is preferred as lesser bit representation would need a lesser cycle for memory fetching and faster computation. Therefore, we need to perform a quantization process to re-encode the information to reduce the representation from 32 bits to 8 bits. The simplest method is the min-max quantization, where the min/max values of FP32 are mapped to min/max values of INT8, respectively. However, the min/max method leads to a significant accuracy loss. Therefore, instead of choosing the min/max value of FP32, they suggest to choose an appropriate threshold T to map from $-T\mathrm{t}\mathrm{o}T$ in FP32 to $-128\mathrm{t}\mathrm{o}127$ in INT8. Figure 9 shows the difference between min/max quantization and threshold quantization. The intuition to choose the value T which minimizes the loss of information between FP32 and INT8. The information loss can be considered as the “Relative Entropy” of two encodings and measured by Kullback-Leibler divergence (KL-divergence). To minimize the loss, they provide the calibration step to interpret a subset of the dataset to find the configuration.

After building the TensorRT engine for an optimized model, we execute the engine using Python APIs. The engine supports synchronous and asynchronous execution, profiling, enumeration, and querying of the bindings for inputs and outputs.

Performance evaluation

We evaluate the performance of the models which are optimized with TensorRT on the Jetson board under various optimzation profiles. Table 7 reports the performance of the eGAC3D model which uses DLA-34 architecture (Yu et al., 2018) as the backbone feature extractor. Additionally, we also develop a lightweight version of eGAC3D named eGAC3D-Lite that adopts ResNet-18 backbone (He et al., 2016). Table 8 summarises the performance of eGAC3D-Lite version. The comparison metrics include model size, inference time (not include pre/postprocessing time), the accuracy of 3D detection $\left(\text{AP}{|}_{R40}^{3D}\right)$, and Bird-Eye-View $\left(\text{AP}{|}_{R40}^{BEV}\right)$. The first two rows show the original Pytorch model’s performance when running on the mainframe computer with RTX-2080ti and the Jetson Xavier NX, respectively. The other three rows present the performance of the optimized models with TensorRT in three different precision modes.

It can be seen from Tables 7 and 8 that there is a severe gap in the latency when deploying the original Pytorch model on the Jetson board. The inference time escalates from 34 to 491.6 ms (for DLA-34 backbone) and from 14 to 209 ms (for ResNet-18 backbone). Although running with Jetson’s GPU, the latency is still tremendous, and it is hard to apply it to real-time applications.

On the other hand, with the support of TensorRT, the inference time is significantly reduced. For the DLA-34 model, with the FP32 precision, the same precision when training the model, the TensorRT engine can run in just 245 ms, speeds up $2\times$ times. If we quantize the precision to FP16 mode, the latency can reduce more than $3\times$ times to 161 ms. For the ResNet-18 lightweight model, the efficiency of TensorRT optimization is more significant. The inference time can decrease up to $10\times$ times, from 209 ms to 88.8 (FP32 mode) and 22.65 (FP16 mode), achieving real-time inference. INT8 mode gives the fastest results, but they are not noticeably improved when compared with FP16. The model’s size can also be reduced with FP16 and INT8 mode. Figure 10 exhibits the throughput (in FPS) based on run-time results of the eGAC3D and eGAC3D-Lite models using Pytorch and TensorRT inference on Jetson Xavier NX.

The FP32 and FP16 engines almost achieve the same performance as the original model in terms of accuracy. In INT8 precision mode, due to the information loss when quantizing the activation outputs of the model’s layers, the model’s accuracy is lower than the original model. As our model is used for detection task, the activation outputs of its layers, especially the final layers in the regression heads, are variant and sensitive. Therefore, quantizing from FP32 to INT8 leads to a significant gap in the model’s final outputs then affects the accuracy.