Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
Skip to main content
Springer Nature Link
Log in

CenRadfusion: fusing image center detection and millimeter wave radar for 3D object detection

  • Original Paper
  • Published:
Signal, Image and Video Processing Aims and scope Submit manuscript

Abstract

The fusion of visual and millimeter-wave radar data has emerged as a prominent solution for precise 3D object detection. This paper focuses on the fusion of visual and mmWave radar information and presents an enhanced fusion method called CenRadfusion. This method represents an evolution and improvement over the classic CenterFusion network by leveraging the fused features from mmWave radar and camera data to achieve accurate 3D object detection. The key features of this method are as follows:To ensure the integrity of the fusion architecture, mmWave radar point clouds are initially projected onto the image plane and added as an additional channel to the input of the CenterNet image detection network. This process forms preliminary 3D detection boxes.Subsequently, mmWave radar point clouds are subjected to density-based clustering, which results in the acquisition of labels and the elimination of irrelevant point clouds and white noise. This step enhances data quality and the reliability of object detection.Finally, an attention module, known as the Squeeze-and-Excitation Networks, is incorporated to weight each feature channel, thereby enhancing the importance of crucial features in the network.Experimental results demonstrate that compared to the original CenterFusion algorithm, the detection Average Precision (AP) values for cars, trucks, and motorcycles have improved by 7.8%, 5.5%, and 5.4%, respectively.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
$34.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or eBook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7

Similar content being viewed by others

Data availability

No new data were created in this study. Data sharing is not applicable to this article.

References

  1. Qian, R., Lai, X., Li, X.: 3d object detection for autonomous driving: a survey. Pattern Recogn. 130, 108796 (2022)

    Article  Google Scholar 

  2. Mousavian, A., Anguelov, D., Flynn, J., et al.: 3d bounding box estimation using deep learning and geometry[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. (2017): 7074–7082.

  3. Liu, Z., Wu, Z., Tóth, R.,: Smoke: Single-stage monocular 3d object detection via keypoint estimation[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops. (2020): 996–997.

  4. Wang, T,, Zhu, X,, Pang, J., et al.: Fcos3d: Fully convolutional one-stage monocular 3d object detection//Proceedings of the IEEE/CVF international conference on computer vision. 2021: 913–922.

  5. Aziz, K., Greef, E.D., Rykunov, M., Bourdoux, A., Sahli, H. Radar-camera Fusion for Road Target Classification. In Proceedings of the 2020 IEEE radar conference (RadarConf20), Florence, Italy, 21–27 September (2020).

  6. Chang, S., Zhang, Y., Zhang, F., Zhao, X., Huang, S., Feng, Z., Wei, Z.: Spatial attention fusion for obstacle detection using mmwave radar and vision sensor. Sensors 20, 956 (2020)

    Article  Google Scholar 

  7. John, V., Mita, S.: Deep feature-level sensor fusion using skip connections for real-time object detection in autonomous driving. Electronics 10, 424 (2021)

    Article  Google Scholar 

  8. Nabati and Qi, H.: Centerfusion: Center-based radar and camera fusion for 3d object detection, In 2021 IEEE winter conference on applications of computer vision (WACV), 2021, pp. 1526–1535

  9. Ester, Martin, et al. "A density-based algorithm for discovering clusters in large spatial databases with noise." kdd. Vol. 96. No. 34. (1996).

  10. Wang, Y., Guan, Y., Li, S., et al.: Fusion perception of vision and millimeter wave radar for autonomous driving//Proceedings of the 8th international conference on computing and artificial intelligence. (2022) pp 767–772.

  11. Hu, J., Shen, L., Sun, G.: Squeeze-and-excitation networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132–7141. (2018) USA.

  12. Holger, C., Varun, B., Alex H., L., Sourabh, V.,Venice Erin, L., Qiang, X., Anush, K., Yu, P.,Giancarlo, B., and Oscar, B., nuscenes: A multimodal dataset for autonomous driving. arXiv preprintarXiv:1903.11027, (2019).

  13. Ji, Z., Prokhorov, D., Radar-vision fusion for object classification. In Proceedings of the automation congress, Cologne, Germany, 30 June–3 July (2008).

  14. Han, S., Xiao, W., Xu, L., Sun, H., Zheng, N. Frontal object perception for intelligent vehicles based on radar and camera fusion. In Proceedings of the control conference, Chengdu, China, pp 27–29 July (2016).

  15. Zeng, S., Zhang, W., Litkouhi, B.B.: Fusion of obstacle detection using radar and camera. US09429650B2, 30 August (2016).

  16. Jin, L., Fu, M.Y., Wang, M.L., Yang, Y.: Vehicle detection based on vision and millimeter wave radar. J. Infrared Millim. Waves 33, 465–471 (2014)

    Google Scholar 

  17. Song, W., Yi, Y., Fu, M., Fan, Q., Wang, M.: Real-time obstacles detection and status classification for collision warning in a vehicle active safety system. IEEE Trans. Intell. Transp. Syst. 19, 758–773 (2018)

    Article  Google Scholar 

  18. John, V., Mita, S.: RVNet: Deep Sensor Fusion of Monocular Camera and Radar for image-Based Obstacle Detection in Challenging Environments. Springer, Cham, Switzerland (2019)

    Google Scholar 

  19. Kocic, J., Jovii, N., Drndarevic, V., Sensors and sensor fusion in autonomous vehicles. In Proceedings of the 2018 26th telecommunications forum (TELFOR), Belgrade, Serbia, 20–21 (2018); pp. 420–425.

  20. Kim, K.E.; Lee, C.J.; Pae, D.S.; Lim, M.T. Sensor fusion for vehicle tracking with camera and radar sensor. In Proceedings of the 2017 17th International Conference on Control, Automation and Systems (ICCAS), Jeju, Korea, 18–21 October 2017.

  21. Lekic, V., Babic, Z.: Automotive radar and camera fusion using Generative Adversarial Networks. Comput. Vis. Image Underst. 184, 1–8 (2019)

    Article  Google Scholar 

  22. Xingyi, Z., Dequan, W., and Philipp K.: Objects as points. arXiv preprint arXiv:1904.07850, 2019.

  23. Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: unified, real-time object detection. In: CVPR (2016)

  24. Liu, W., Anguelov, D., Erhan, D., et al. Ssd: Single shot multibox detector//Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14. Springer International Publishing, (2016): 21-37.

  25. Girshick, R.,: Fast r-cnn[C]//Proceedings of the IEEE international conference on computer vision. (2015) pp 1440–1448.

  26. Tao, W., Zheng, N., Xin, J., Ma, Z.: Integrating millimeter wave radar with a monocular vision sensor for on-road obstacle detection applications. Sensors 11, 8992–9008 (2011)

    Article  Google Scholar 

  27. Shi, X., Ye, Q., Chen, X., et al.: Geometry-based distance decomposition for monocular 3d object detection[C]//Proceedings of the IEEE/CVF international conference on computer vision. (2021) pp 15172–15181.

  28. Bhattacharyya P., Chengjie H., and Krzysztof C.: Sa-det3d: Self-attention based context-aware 3d object detection. Proceedings of the IEEE/CVF international conference on computer vision. (2021).

  29. Svenningsson., P, Fioranelli, F., Yarovoy, A.,: Radar-pointgnn: Graph based object recognition for unstructured radar point-cloud data[C]//2021 IEEE Radar Conference (RadarConf21). IEEE, (2021), pp 1–6.

Download references

Funding

This work was funded by the Natural Science Foundation of AnhuiProvince (2208085MF173)and the Joint Research Project of Yangtze River Delta Science and Technology Innovation Community (2023CSJGG1600)and the Major Science and Technology Project of "Red Casting Light" in Wuhu City (2023zd01, 2023zd03).

Author information

Author notes

  1. Aixi Yang and Zhiqiang Liu have contributed equally to this work.

Authors and Affiliations

  1. School of Mechanical And Automotive Engineering, Anhui Polytechnic University, Wuhu, 241000, China

    Peicheng Shi, Tong Jiang & Zhiqiang Liu

  2. Department Polytechnic Institute of Zhejiang University, Hangzhou, 310000, China

    Aixi Yang

Authors
  1. Peicheng Shi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Tong Jiang
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Aixi Yang
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Zhiqiang Liu
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

Peicheng Shi: Conceptualization; Funding acquisition; Project administration; Data curation; Writing—review and editing. Tong Jiang: Software, Methodology; Resources; Writing original draft preparation, Visualization. Aixi Yang: Supervision; Validation. Zhiqiang Liu: Formal analysis; Investigation. All authors have read and agreed to the published version of the manuscript.

Corresponding author

Correspondence to Peicheng Shi.

Ethics declarations

Conflict of interest

The authors declare no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Shi, P., Jiang, T., Yang, A. et al. CenRadfusion: fusing image center detection and millimeter wave radar for 3D object detection. SIViP 18, 5811–5821 (2024). https://doi.org/10.1007/s11760-024-03273-3

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s11760-024-03273-3

Keywords

Search

Navigation