Abstract
The tasks of object detection and trajectory forecasting play a crucial role in understanding the scene for autonomous driving. These tasks are typically executed in a cascading manner, making them prone to compounding errors. Furthermore, there is usually a very thin interface between the two tasks, creating a lossy information bottleneck. To address these challenges, our approach formulates the union of the two tasks as a trajectory refinement problem, where the first pose is the detection (current time), and the subsequent poses are the waypoints of the multiple forecasts (future time). To tackle this unified task, we design a refinement transformer that infers the presence, pose, and multi-modal future behaviors of objects directly from LiDAR point clouds and high-definition maps. We call this model DeTra, short for object Detection and Trajectory forecasting. In our experiments, we observe that DeTra outperforms the state-of-the-art on Argoverse 2 Sensor and Waymo Open Dataset by a large margin, across a broad range of metrics. Finally, we perform extensive ablation studies that show the value of refinement for this task and that key design choices were made.
S. Casas, B. Agro and J. Mao—Equal contribution.
J. Mao and A. Cui—Work done while at Waabi.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
References
Agro, B., Sykora, Q., Casas, S., Urtasun, R.: Implicit occupancy flow fields for perception and prediction in self-driving. In: CVPR (2023)
Cai, Z., Vasconcelos, N.: Cascade r-cnn: high quality object detection and instance segmentation. PAMI (2019)
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., Zagoruyko, S.: End-to-end object detection with transformers. In: ECCV (2020)
Casas, S., Gulino, C., Liao, R., Urtasun, R.: Spagnn: spatially-aware graph neural networks for relational behavior forecasting from sensor data. In: ICRA (2020)
Casas, S., Gulino, C., Suo, S., Luo, K., Liao, R., Urtasun, R.: Implicit latent variable model for scene-consistent motion forecasting. In: ECCV (2020)
Casas, S., Luo, W., Urtasun, R.: Intentnet: learning to predict intention from raw sensor data. In: CoRL (2018)
Casas, S., Sadat, A., Urtasun, R.: Mp3: a unified model to map, perceive, predict and plan. In: CVPR (2021)
Chai, Y., Sapp, B., Bansal, M., Anguelov, D.: Multipath: multiple probabilistic anchor trajectory hypotheses for behavior prediction. arXiv preprint arXiv:1910.05449 (2019)
Chen, Y., Dai, X., Liu, M., Chen, D., Yuan, L., Liu, Z.: Dynamic convolution: attention over convolution kernels. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11030–11039 (2020)
Chitta, K., Prakash, A., Jaeger, B., Yu, Z., Renz, K., Geiger, A.: Transfuser: Imitation with transformer-based sensor fusion for autonomous driving. PAMI (2022)
Cho, K., et al.: Learning phrase representations using rnn encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078 (2014)
Cui, A., Casas, S., Sadat, A., Liao, R., Urtasun, R.: Lookout: diverse multi-future prediction and planning for self-driving. In: ICCV (2021)
Cui, A., Casas, S., Wong, K., Suo, S., Urtasun, R.: Gorela: go relative for viewpoint-invariant motion forecasting. arXiv preprint arXiv:2211.02545 (2022)
Cui, H., et al.: Multimodal trajectory predictions for autonomous driving using deep convolutional networks. 2019 ICRA, pp. 2090–2096 (2018). https://api.semanticscholar.org/CorpusID:52891221
Deo, N., Wolff, E.M., Beijbom, O.: Multimodal trajectory prediction conditioned on lane-graph traversals. In: CoRL (2021)
Djuric, N., et al.: Multixnet: multiclass multistage multimodal motion prediction. IV (2021)
Ettinger, S., et al.: Large scale interactive motion forecasting for autonomous driving: The waymo open motion dataset. In: ICCV (2021)
Fan, H., et al.: Baidu apollo em motion planner. arXiv preprint arXiv:1807.08048 (2018)
Gao, J., et al.: Vectornet: encoding hd maps and agent dynamics from vectorized representation. CVPR (2020)
Gilles, T., Sabatini, S., Tsishkou, D., Stanciulescu, B., Moutarde, F.: Thomas: trajectory heatmap output with learned multi-agent sampling. arXiv preprint arXiv:2110.06607 (2021)
Girgis, R., et al.: Latent variable sequential set transformers for joint multi-agent motion prediction. arXiv preprint arXiv:2104.00563 (2021)
Grubb, A., Bagnell, D.: Speedboost: Anytime prediction with uniform near-optimality. In: Artificial Intelligence and Statistics, pp. 458–466. PMLR (2012)
He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: CVPR, pp. 770–778 (2016)
Hu, H., Dey, D., Hebert, M., Bagnell, J.A.: Learning anytime predictions in neural networks via adaptive loss balancing. In: Proceedings of the AAAI Conference on Artificial Intelligence (2019)
Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7132–7141 (2018)
Hu, Y., et al.: Planning-oriented autonomous driving. In: CVPR (2023)
Ivanovic, B., Lin, Y., Shrivastava, S., Chakravarty, P., Pavone, M.: Propagating state uncertainty through trajectory forecasting. In: ICRA (2022)
Kipf, T.N., Welling, M.: Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 (2016)
Kumbhar, O., Sizikova, E., Majaj, N., Pelli, D.G.: Anytime prediction as a model of human reaction time. arXiv preprint arXiv:2011.12859 (2020)
Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: Fast encoders for object detection from point clouds. CVPR (2018)
Li, L.L., et al.: End-to-end contextual perception and prediction with interaction transformer. IROS (2020)
Liang, M., et al.: Learning lane graph representations for motion forecasting. In: Vedaldi, A., Bischof, H., Brox, T., Frahm, J.-M. (eds.) ECCV 2020. LNCS, vol. 12347, pp. 541–556. Springer, Cham (2020). https://doi.org/10.1007/978-3-030-58536-5_32
Liang, M., et al.: Pnpnet: end-to-end perception and prediction with tracking in the loop. In: CVPR (2020)
Lin, T.Y., Goyal, P., Girshick, R., He, K., Dollár, P.: Focal loss for dense object detection. In: ICCV (2017)
Liu, S., et al.: DAB-DETR: Dynamic anchor boxes are better queries for DETR. In: ICLR (2022)
Liu, W., et al.: Ssd: single shot multibox detector. In: ECCV (2015)
Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)
Luo, W., Yang, B., Urtasun, R.: Fast and furious: real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net. In: CVPR (2018)
Mahjourian, R., Kim, J., Chai, Y., Tan, M., Sapp, B., Anguelov, D.: Occupancy flow fields for motion forecasting in autonomous driving. IEEE Robot. Autom. Lett. 7(2), 5639–5646 (2022)
Meyer, G.P., et al.: Laserflow: efficient and probabilistic object detection and motion forecasting. IEEE Robot. Autom. Lett. 6, 526–533 (2021)
Mo, X., Huang, Z., Xing, Y., Lv, C.: Multi-agent trajectory prediction with heterogeneous edge-enhanced graph attention network. IEEE Trans. Intell. Transp. Syst. (2022)
Nayakanti, N., Al-Rfou, R., Zhou, A., Goel, K., Refaat, K.S., Sapp, B.: Wayformer: Motion forecasting via simple & efficient attention networks. arXiv preprint arXiv:2207.05844 (2022)
Ngiam, J., et al.: Scene transformer: A unified architecture for predicting multiple agent trajectories. arXiv preprint arXiv:2106.08417 (2021)
Phan-Minh, T., Grigore, E.C., Boulton, F.A., Beijbom, O., Wolff, E.M.: Covernet: Multimodal behavior prediction using trajectory sets. CVPR (2019)
Qi, C.R., Su, H., Mo, K., Guibas, L.J.: Pointnet: Deep learning on point sets for 3d classification and segmentation. In: CVPR (2017)
Qi, C.R., et al.: Offboard 3d object detection from point cloud sequences. In: CVPR (2021)
Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: Unified, real-time object detection. In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) (2015)
Ren, S., He, K., Girshick, R.B., Sun, J.: Faster r-cnn: towards real-time object detection with region proposal networks. PAMI (2015)
Rhinehart, N., McAllister, R., Levine, S.: Deep imitative models for flexible inference, planning, and control. arXiv preprint arXiv:1810.06544 (2018)
Sadat, A., Ren, M., Pokrovsky, A., Lin, Y.C., Yumer, E., Urtasun, R.: Jointly learnable behavior and trajectory planning for self-driving vehicles. In: 2019 IROS, pp. 3949–3956. IEEE (2019)
Shah, M., et al.: Liranet: end-to-end trajectory prediction using spatio-temporal radar fusion. In: CoRL (2020)
Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: CVPR (2020)
Varadarajan, B., et al.: Multipath++: Efficient information fusion and trajectory aggregation for behavior prediction. In: 2022 ICRA (2022)
Vaswani, A., et al.: Attention is all you need. NeurIPS (2017)
Weng, X., Ivanovic, B., Pavone, M.: Mtp: multi-hypothesis tracking and prediction for reduced error propagation. In: IV (2022)
Wilson, B., et al.: Argoverse 2: next generation datasets for self-driving perception and forecasting. arXiv preprint arXiv:2301.00493 (2023)
Yang, A.J., et al.: Labelformer: object trajectory refinement for offboard perception from lidar point clouds. In: CoRL. PMLR (2023)
Yang, B., Bai, M., Liang, M., Zeng, W., Urtasun, R.: Auto4d: learning to label 4d objects from sequential point clouds. arXiv preprint arXiv:2101.06586 (2021)
Yang, B., Luo, W., Urtasun, R.: Pixor: real-time 3d object detection from point clouds. In: CVPR (2018)
Yuan, Y., Weng, X., Ou, Y., Kitani, K.M.: Agentformer: agent-aware transformers for socio-temporal multi-agent forecasting. In: ICCV (2021)
Zeng, W., et al.: End-to-end interpretable neural motion planner. In: CVPR (2019)
Zhou, X., Wang, D., Krähenbühl, P.: Objects as points. ArXiv (2019)
Zhou, Y., et al.: End-to-end multi-view fusion for 3d object detection in lidar point clouds. ArXiv (2019)
Zhou, Y., Tuzel, O.: Voxelnet: end-to-end learning for point cloud based 3d object detection. In: 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2017)
Zhou, Z., Wang, J., Li, Y.H., Huang, Y.K.: Query-centric trajectory prediction. In: CVPR (2023)
Zhou, Z., Ye, L., Wang, J., Wu, K., Lu, K.: Hivt: hierarchical vector transformer for multi-agent motion prediction. In: CVPR (2022)
Zhu, X., Su, W., Lu, L., Li, B., Wang, X., Dai, J.: Deformable detr: deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04159 (2020)
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
1 Electronic supplementary material
Below is the link to the electronic supplementary material.
Rights and permissions
Copyright information
© 2025 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Casas, S. et al. (2025). DeTra: A Unified Model for Object Detection and Trajectory Forecasting. In: Leonardis, A., Ricci, E., Roth, S., Russakovsky, O., Sattler, T., Varol, G. (eds) Computer Vision – ECCV 2024. ECCV 2024. Lecture Notes in Computer Science, vol 15146. Springer, Cham. https://doi.org/10.1007/978-3-031-73223-2_19
Download citation
DOI: https://doi.org/10.1007/978-3-031-73223-2_19
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-73222-5
Online ISBN: 978-3-031-73223-2
eBook Packages: Computer ScienceComputer Science (R0)