V2X-Real: A Largs-Scale Dataset for Vehicle-to-Everything Cooperative Perception
Authors: 
Hao Xiang, Zhaoliang Zheng, Xin Xia, Runsheng Xu, Letian Gao, Zewei Zhou, Xu Han, Xinkai Ji, Mingxi Li, Zonglin Meng, Li Jin, Mingyue Lei, Zhaoyang Ma, Zihang He, Haoxuan Ma, Yunshuang Yuan, Yingqian Zhao, Jiaqi MaAuthors Info & Claims
Hao Xiang, Zhaoliang Zheng, Xin Xia, Runsheng Xu, Letian Gao, Zewei Zhou, Xu Han, Xinkai Ji, Mingxi Li, Zonglin Meng, Li Jin, Mingyue Lei, Zhaoyang Ma, Zihang He, Haoxuan Ma, Yunshuang Yuan, Yingqian Zhao, Jiaqi MaAuthors Info & ClaimsPages 455 - 470
Abstract
Recent advancements in Vehicle-to-Everything (V2X) technologies have enabled autonomous vehicles to share sensing information to see through occlusions, greatly boosting the perception capability. However, there are no real-world datasets to facilitate the real V2X cooperative perception research – existing datasets either only support Vehicle-to-Infrastructure cooperation or Vehicle-to-Vehicle cooperation. In this paper, we present V2X-Real, a large-scale dataset that includes a mixture of multiple vehicles and smart infrastructure to facilitate the V2X cooperative perception development with multi-modality sensing data. Our V2X-Real is collected using two connected automated vehicles and two smart infrastructure, which are all equipped with multi-modal sensors including LiDAR sensors and multi-view cameras. The whole dataset contains 33K LiDAR frames and 171K camera data with over 1.2 M annotated bounding boxes of 10 categories in very challenging urban scenarios. According to the collaboration mode and ego perspective, we derive four types of datasets for Vehicle-Centric, Infrastructure-Centric, Vehicle-to-Vehicle, and Infrastructure-to-Infrastructure cooperative perception. Comprehensive multi-class multi-agent benchmarks of SOTA cooperative perception methods are provided. The V2X-Real dataset and codebase are available at https://mobility-lab.seas.ucla.edu/v2x-real.
References
[1]
Caesar, H., et al.: nuScenes: a multimodal dataset for autonomous driving. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11621–11631 (2020)
[2]
Chen, Q., Ma, X., Tang, S., Guo, J., Yang, Q., Fu, S.: F-cooper: Feature based cooperative perception for autonomous vehicle edge computing system using 3D point clouds. In: Proceedings of the 4th ACM/IEEE Symposium on Edge Computing, pp. 88–100 (2019)
[3]
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., Koltun, V.: CARLA: an open urban driving simulator. In: Conference on Robot Learning, pp. 1–16. PMLR (2017)
[4]
Gao L, Xia X, Zheng Z, and Ma J GNSS/IMU/LiDAR fusion for vehicle localization in urban driving environments within a consensus framework Mech. Syst. Signal Process. 2023 205 110862
[5]
Geiger, A., Lenz, P., Urtasun, R.: Are we ready for autonomous driving? The kitti vision benchmark suite. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3354–3361. IEEE (2012)
[6]
Hao, R., et al.: RCooper: a real-world large-scale dataset for roadside cooperative perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22347–22357 (2024)
[7]
Hu, Y., Fang, S., Lei, Z., Zhong, Y., Chen, S.: Where2comm: communication-efficient collaborative perception via spatial confidence maps. In: Advances in Neural Information Processing Systems, vol. 35, pp. 4874–4886 (2022)
[8]
Huang Y, Du J, Yang Z, Zhou Z, Zhang L, and Chen H A survey on trajectory-prediction methods for autonomous driving IEEE Trans. Intell. Vehicles 2022 7 3 652-674
[9]
Lang, A.H., Vora, S., Caesar, H., Zhou, L., Yang, J., Beijbom, O.: Pointpillars: fast encoders for object detection from point clouds. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12697–12705 (2019)
[10]
Li, E., Wang, S., Li, C., Li, D., Wu, X., Hao, Q.: Sustech points: a portable 3D point cloud interactive annotation platform system. In: 2020 IEEE Intelligent Vehicles Symposium (IV), pp. 1108–1115. IEEE (2020)
[11]
Li, Y., Zhao, S.Z., Xu, C., Tang, C., Li, C., Ding, M., Tomizuka, M., Zhan, W.: Pre-training on synthetic driving data for trajectory prediction (2023)
[12]
Li Y et al. V2x-sim: multi-agent collaborative perception dataset and benchmark for autonomous driving IEEE Robot. Autom. Lett. 2022 7 4 10914-10921
[13]
Li, Y., Ren, S., Wu, P., Chen, S., Feng, C., Zhang, W.: Learning distilled collaboration graph for multi-agent perception. In: Advances in Neural Information Processing Systems, vol. 34, pp. 29541–29552 (2021)
[14]
Liao Y, Xie J, and Geiger A Kitti-360: a novel dataset and benchmarks for urban scene understanding in 2D and 3D IEEE Trans. Pattern Anal. Mach. Intell. 2022 45 3 3292-3310
[15]
Lu, Y., Hu, Y., Zhong, Y., Wang, D., Chen, S., Wang, Y.: An extensible framework for open heterogeneous collaborative perception. arXiv preprint arXiv:2401.13964 (2024)
[16]
Lu, Y., et al.: Robust collaborative 3D object detection in presence of pose errors. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 4812–4818. IEEE (2023)
[17]
Rawashdeh, Z.Y., Wang, Z.: Collaborative automated driving: a machine learning-based method to enhance the accuracy of shared information. In: 2018 21st International Conference on Intelligent Transportation Systems (ITSC), pp. 3961–3966. IEEE (2018)
[18]
Su, S., et al.: Uncertainty quantification of collaborative detection for self-driving. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 5588–5594. IEEE (2023)
[19]
Sun, P., et al.: Scalability in perception for autonomous driving: Waymo open dataset. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2446–2454 (2020)
[20]
Team, O.D.: OpenPCDet: an open-source toolbox for 3D object detection from point clouds (2020). https://github.com/open-mmlab/OpenPCDet
[21]
Wang T-H, Manivasagam S, Liang M, Yang B, Zeng W, and Urtasun R Vedaldi A, Bischof H, Brox T, and Frahm J-M V2VNet: vehicle-to-vehicle communication for joint perception and prediction Computer Vision – ECCV 2020 2020 Cham Springer 605-621
[22]
Wei, S., et al.: Asynchrony-robust collaborative perception via bird’s eye view flow. In: Advances in Neural Information Processing Systems (2023)
[23]
Wu, Z., Wang, Y., Ma, H., Li, Z., Qiu, H., Li, J.: CMP: cooperative motion prediction with multi-agent communication. arXiv preprint arXiv:2403.17916 (2024)
[24]
Xia X et al. An automated driving systems data acquisition and analytics platform Transp. Res. Part C Emerg. Technol. 2023 151 104120
[25]
Xiang, H., Xu, R., Ma, J.: HM-ViT: hetero-modal vehicle-to-vehicle cooperative perception with vision transformer. arXiv preprint arXiv:2304.10628 (2023)
[26]
Xiang, H., Xu, R., Xia, X., Zheng, Z., Zhou, B., Ma, J.: V2XP-ASG: generating adversarial scenes for vehicle-to-everything perception. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 3584–3591. IEEE (2023)
[27]
Xu, C., et al.: PreTraM: self-supervised pre-training via connecting trajectory and map. arXiv preprint arXiv:2204.10435 (2022)
[28]
Xu, R., Chen, W., Xiang, H., Xia, X., Liu, L., Ma, J.: Model-agnostic multi-agent perception framework. In: 2023 IEEE International Conference on Robotics and Automation (ICRA), pp. 1471–1478. IEEE (2023)
[29]
Xu, R., Guo, Y., Han, X., Xia, X., Xiang, H., Ma, J.: OpenCDA: an open cooperative driving automation framework integrated with co-simulation. In: 2021 IEEE International Intelligent Transportation Systems Conference (ITSC), pp. 1155–1162. IEEE (2021)
[30]
Xu, R., Tu, Z., Xiang, H., Shao, W., Zhou, B., Ma, J.: CoBEVT: cooperative bird’s eye view semantic segmentation with sparse transformers. arXiv preprint arXiv:2207.02202 (2022)
[31]
Xu, R., et al.: V2V4Real: a real-world large-scale dataset for vehicle-to-vehicle cooperative perception. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13712–13722 (2023)
[32]
Xu, R., et al.: The OpenCDA open-source ecosystem for cooperative driving automation research. IEEE Trans. Intell. Veh. (2023)
[33]
Xu R, Xiang H, Tu Z, Xia X, Yang MH, and Ma J Avidan S, Brostow G, Cissé M, Farinella GM, and Hassner T V2X-ViT: vehicle-to-everything cooperative perception with vision transformer Computer Vision – ECCV 2022 2022 Cham Springer 107-124
[34]
Xu, R., Xiang, H., Xia, X., Han, X., Li, J., Ma, J.: OPV2V: an open benchmark dataset and fusion pipeline for perception with vehicle-to-vehicle communication. In: 2022 International Conference on Robotics and Automation (ICRA), pp. 2583–2589. IEEE (2022)
[35]
Yu, H., et al.: DAIR-v2x: a large-scale dataset for vehicle-infrastructure cooperative 3D object detection. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 21361–21370 (2022)
[36]
Zheng, Z., Han, X., Xia, X., Gao, L., Xiang, H., Ma, J.: OpenCDA-ROS: enabling seamless integration of simulation and real-world cooperative driving automation. IEEE Trans. Intell. Veh. (2023)
[37]
Zhou, Y., Tuzel, O.: VoxelNet: end-to-end learning for point cloud based 3D object detection, pp. 4490–4499 (2018)
[38]
Zhou, Z., Yang, Z., Zhang, Y., Huang, Y., Chen, H., Yu, Z.: A comprehensive study of speed prediction in transportation system: from vehicle to traffic. Iscience 25(3) (2022)
Index Terms
- V2X-Real: A Largs-Scale Dataset for Vehicle-to-Everything Cooperative Perception
Index terms have been assigned to the content through auto-classification.
Recommendations
V2X-ViT: Vehicle-to-Everything Cooperative Perception with Vision Transformer
Computer Vision – ECCV 2022AbstractIn this paper, we investigate the application of Vehicle-to-Everything (V2X) communication to improve the perception performance of autonomous vehicles. We present a robust cooperative perception framework with V2X communication using a novel ...
Multivehicle Cooperative Driving Using Cooperative Perception: Design and Experimental Validation
In this paper, we present a multivehicle cooperative driving system architecture using cooperative perception along with experimental validation. For this goal, we first propose a multimodal cooperative perception system that provides see-through, lifted-...
Safe Crossroads via Vehicle to Vehicle Communication
Computer Aided Systems Theory - EUROCAST 2009Driving through crossroads is one of the most dangerous maneuvers. The European community goal of reducing the vehicle accident rate will require a reduction of accidents in crossroads. This paper presents a method of cooperation among vehicles getting ...
Comments
Information & Contributors
Information
Published In
Sep 2024
577 pages
ISBN:978-3-031-72942-3
DOI:10.1007/978-3-031-72943-0
© The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.
Publisher
Springer-Verlag
Berlin, Heidelberg
Publication History
Published: 29 November 2024
Author Tags
Qualifiers
- Article
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 0Total Downloads
- Downloads (Last 12 months)0
- Downloads (Last 6 weeks)0
Reflects downloads up to 07 Mar 2025