research-article

Multi-Dimensional and Message-Guided Fuzzing for Robotic Programs in Robot Operating System

Authors:

Shi-Min HuAuthors Info & Claims

ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2

Pages 763 - 778

https://doi.org/10.1145/3620665.3640425

Published: 27 April 2024 Publication History

Abstract

An increasing number of robotic programs are implemented based on Robot Operating System (ROS), which provides many practical tools and libraries for robot development. To improve robot reliability and security, several recent approaches apply fuzzing to ROS programs for bug detection. However, these approaches still have some main limitations, including inefficient test case generation, ineffective program feedback and weak generality/automation.

In this paper, we design a new fuzzing framework named ROFER, to effectively test robotic programs in ROS for bug detection. Compared to existing ROS fuzzing approaches, ROFER has two novel techniques: (1) a dimension-level mutation method that considers the contribution of each input dimension to testing coverage, to generate efficient test cases from multiple dimensions; (2) a message-guided fuzzing approach that uses a new coverage metric named message feature, to reflect the robot's possible state transitions affected by multiple ROS nodes. We evaluate ROFER on 13 common robotic programs in ROS2, and it finds 88 real bugs, 46 of which have been confirmed by ROS developers. We compare ROFER to four state-of-the-art ROS fuzzing approaches, and it finds more bugs with higher testing coverage.

References

[1]

American Fuzzy Lop. http://lcamtuf.coredump.cx/afl/.

[2]

ASan: address sanitizer. https://github.com/google/sanitizers/wiki/AddressSanitizer.

[3]

AutoCarROS2: autonomous vehicle control and behaviour. https://github.com/winstxnhdw/AutoCarROS2.

[4]

Jinsheng Ba, Gregory J Duck, and Abhik Roychoudhury. Efficient greybox fuzzing to detect memory errors. In Proceedings of the 37th International Conference on Automated Software Engineering (ASE), pages 1--12, 2022.

Digital Library

[5]

Jia-Ju Bai, Yu-Ping Wang, Jie Yin, and Shi-Min Hu. Testing error handling code in device drivers using characteristic fault injection. In Proceedings of the 2016 USENIX Annual Technical Conference, pages 635--647, 2016.

Digital Library

[6]

Tim Blazytko, Cornelius Aschermann, Moritz Schlögel, Ali Abbasi, Sergej Schumilo, Simon Wörner, and Thorsten Holz. GRIMOIRE: synthesizing structure while fuzzing. In Proceedings of the 28th USENIX Security Symposium, volume 19, 2019.

[7]

Mohamed-Slim Bouguerra, Thierry Gautier, Denis Trystram, and Jean-Marc Vincent. A flexible checkpoint/restart model in distributed systems. In Proceedings of the 8th International Conference on Parallel Processing and Applied Mathematics (PPAM), pages 206--215, 2010.

[8]

Anton Burtsev, Prashanth Radhakrishnan, Mike Hibler, and Jay Lepreau. Transparent checkpoints of closed distributed systems in emulab. In Proceedings of the 4th European Conference on Computer Systems (EuroSys), pages 173--186, 2009.

Digital Library

[9]

Cartographer: a system for real-time 2D and 3D SLAM across multiple platforms and sensor configurations. https://github.com/cartographer-project/cartographer_ros.

[10]

Renato Carvalho, Alcino Cunha, Nuno Macedo, and André Santos. Verification of system-wide safety properties of ROS applications. In Proceedings of the 2020 International Conference on Intelligent Robots and Systems (IROS), pages 7249--7254, 2020.

Digital Library

[11]

Hongxu Chen, Shengjian Guo, Yinxing Xue, Yulei Sui, Cen Zhang, Yuekang Li, Haijun Wang, and Yang Liu. MUZZ: thread-aware grey-box fuzzing for effective bug hunting in multithreaded programs. In Proceedings of the 29th USENIX Security Symposium, pages 2325--2342, 2020.

[12]

Common messages in ROS. http://wiki.ros.org/common_msgs.

[13]

Kai Cong, Li Lei, Zhenkun Yang, and Fei Xie. Automatic fault injection for driver robustness testing. In Proceedings of the 2015 International Symposium on Software Testing and Analysis (ISSTA), pages 361--372, 2015.

Digital Library

[14]

Jin Cui, Cong Tian, Nan Zhang, Zhenhua Duan, and Hongwei Du. Verifying schedulability of tasks in ROS-based systems. Journal of Combinatorial Optimization, 37:901--920, 2019.

Digital Library

[15]

Rodrigo Delgado, Miguel Campusano, and Alexandre Bergel. Fuzz testing in behavior-based robotics. In Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), pages 9375--9381, 2021.

Digital Library

[16]

František Duchoň, Jakub Hažík, Jozef Rodina, Michal Tölgyessy, Martin Dekan, and Adam Sojka. Verification of slam methods implemented in ROS. Journal of Multidisciplinary Engineering Science and Technology (JMEST), 6(9):2458--9403, 2019.

[17]

Fuzzing random programs without using execve. https://lcamtuf.blogspot.com/2014/10/fuzzing-binaries-without-execve.html.

[18]

Jingzhou Fu, Jie Liang, Zhiyong Wu, Mingzhe Wang, and Yu Jiang. Griffin: grammar-free DBMS fuzzing. In Proceedings of the 37th International Conference on Automated Software Engineering (ASE), pages 1--12, 2022.

Digital Library

[19]

Shuitao Gan, Chao Zhang, Xiaojun Qin, Xuwen Tu, Kang Li, Zhongyu Pei, and Zuoning Chen. CollAFL: path sensitive fuzzing. In Proceedings of the 39th IEEE Symposium on Security and Privacy, pages 679--696, 2018.

[20]

Gazebo: a robot simulation framework. http://gazebosim.org/.

[21]

Raju Halder, José Proença, Nuno Macedo, and André Santos. Formal verification of ROS-based robotic applications using timed-automata. In Proceedings of the 5th International FME Workshop on Formal Methods in Software Engineering (FormaliSE), pages 44--50, 2017.

Digital Library

[22]

Adrian Herrera, Hendra Gunadi, Shane Magrath, Michael Norrish, Mathias Payer, and Antony L Hosking. Seed selection for successful fuzzing. In Proceedings of the 2021 International Symposium on Software Testing and Analysis (ISSTA), pages 230--243, 2021.

Digital Library

[23]

David Ke Hong, John Kloosterman, Yuqi Jin, Yulong Cao, Qi Alfred Chen, Scott Mahlke, and Z Morley Mao. AVGuardian: detecting and mitigating publish-subscribe overprivilege for autonomous vehicle systems. In Proceedings of the 2020 European Symposium on Security and Privacy (EuroS&P), pages 445--459, 2020.

[24]

Casidhe Hutchison, Milda Zizyte, Patrick E Lanigan, David Guttendorf, Michael Wagner, Claire Le Goues, and Philip Koopman. Robustness testing of autonomy software. In Proceedings of the 40th International Conference on Software Engineering: Software Engineering in Practice Track (ICSE-SEIP), pages 276--285, 2018.

Digital Library

[25]

Janko slavic peak detection method. https://github.com/jankoslavic/py-tools/tree/master/findpeaks.

[26]

Jaka Javh, Janko Slavič, and Miha Boltežar. High frequency modal identification on noisy high-speed camera data. Mechanical Systems and Signal Processing, 98:344--351, 2018.

[27]

Yuseok Jeon, WookHyun Han, Nathan Burow, and Mathias Payer. FuZZan: efficient sanitizer metadata design for fuzzing. In Proceedings of the 2020 USENIX Annual Technical Conference, pages 249--263, 2020.

[28]

Zu-Ming Jiang, Jia-Ju Bai, Kangjie Lu, and Shi-Min Hu. Fuzzing error handling code using context-sensitive software fault injection. In Proceedings of the 29th USENIX Security Symposium, pages 2595--2612, 2020.

Digital Library

[29]

Zu-Ming Jiang, Jia-Ju Bai, Kangjie Lu, and Shi-Min Hu. Context-sensitive and directional concurrency fuzzing for data-race detection. In Proceedings of the 29th Network and Distributed System Security Symposium (NDSS), pages 1--18, 2022.

[30]

Jinho Jung, Hong Hu, Joy Arulraj, Taesoo Kim, and Woonhak Kang. APOLLO: automatic detection and diagnosis of performance regressions in database systems. In Proceedings of the 46th International Conference on Very Large Data Bases (VLDB), pages 57--70, 2020.

[31]

Kalman filter based localization for ROS. https://github.com/rsasaki0109/kalman_filter_localization.

[32]

Deborah S Katz, Casidhe Hutchison, Milda Zizyte, and Claire Le Goues. Detecting execution anomalies as an oracle for autonomy software robustness. In Proceedings of the 2020 International Conference on Robotics and Automation (ICRA), pages 9366--9373, 2020.

[33]

Sean Kauffman, Murray Dunne, Giovani Gracioli, Waleed Khan, Nirmal Benann, and Sebastian Fischmeister. Palisade: A framework for anomaly detection in embedded systems. Journal of Systems Architecture (JSA), 113:101876, 2021.

[34]

Seulbae Kim and Taesoo Kim. RoboFuzz: fuzzing robotic systems over robot operating system (ROS) for finding correctness bugs. In Proceedings of the 30th International Symposium on the Foundations of Software Engineering (FSE), pages 447--458, 2022.

Digital Library

[35]

Seulbae Kim, Meng Xu, Sanidhya Kashyap, Jungyeon Yoon, Wen Xu, and Taesoo Kim. Finding semantic bugs in file systems with an extensible fuzzing framework. In Proceedings of the 27th International Symposium on Operating Systems Principles (SOSP), pages 147--161, 2019.

Digital Library

[36]

George Klees, Andrew Ruef, Benji Cooper, Shiyi Wei, and Michael Hicks. Evaluating fuzz testing. In Proceedings of the 25th International Conference on Computer and Communications Security (CCS), pages 2123--2138, 2018.

Digital Library

[37]

Sitar Kortik and Tejas Kumar Shastha. Formal verification of ROS-based systems using a linear logic theorem prover. In Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), pages 9368--9374, 2021.

Digital Library

[38]

Sofiane Lagraa, Maxime Cailac, Sean Rivera, Frédéric Beck, and Radu State. Real-time attack detection on robot cameras: a self-driving car application. In Proceedings of the 2019 International Conference on Robotic Computing (IRC), pages 102--109, 2019.

[39]

LaMa: alternative localization and mapping. https://github.com/iris-ua/iris_lama_ros.

[40]

Yu Liang, Song Liu, and Hong Hu. Detecting logical bugs of DBMS with coverage-based guidance. In Proceedings of the 31st USENIX Security Symposium, pages 4309--4326, 2022.

[41]

Changming Liu, Deqing Zou, Peng Luo, Bin B Zhu, and Hai Jin. A heuristic framework to detect concurrency vulnerabilities. In Proceedings of the 34th International Annual Computer Security Applications Conference (ACSAC), pages 529--541, 2018.

Digital Library

[42]

llvm-cov: collecting program coverage with llvm framework. https://www.llvm.org/docs/CommandGuide/llvm-cov.html.

[43]

Maze solving using computer vision. https://github.com/HaiderAbasi/ROS2-Path-Planning-and-Maze-Solving.

[44]

Ruijie Meng, George Pîrlea, Abhik Roychoudhury, and Ilya Sergey. Greybox fuzzing of distributed systems. In Proceedings of the 30th International Conference on Computer and Communications Security (CCS), pages 1615--1629, 2023.

Digital Library

[45]

MSan: memory sanitizer. https://github.com/google/sanitizers/wiki/MemorySanitizer.

[46]

Stefan Nagy and Matthew Hicks. Full-speed fuzzing: reducing fuzzing overhead through coverage-guided tracing. In Proceedings of the 40th IEEE Symposium on Security and Privacy, pages 787--802, 2019.

[47]

Roberto Natella. StateAFL: greybox fuzzing for stateful network servers. Empirical Software Engineering, 27(7):1--31, 2022.

Digital Library

[48]

State machine and behavior tree of ROS2 navigation framework. https://navigation.ros.org/behavior_trees/trees/nav_to_pose_recovery.html.

[49]

Burcu Kulahcioglu Ozkan, Rupak Majumdar, and Simin Oraee. Trace aware random testing for distributed systems. Proceedings of the ACM on Programming Languages (OOPSLA), 3(OOPSLA):1--29, 2019.

Digital Library

[50]

Rohan Padhye, Caroline Lemieux, Koushik Sen, Mike Papadakis, and Yves Le Traon. Semantic fuzzing with Zest. In Proceedings of the 2019 International Symposium on Software Testing and Analysis (ISSTA), pages 329--340, 2019.

Digital Library

[51]

Shankara Pailoor, Andrew Aday, and Suman Jana. MoonShine: optimizing OS fuzzer seed selection with trace distillation. In Proceedings of the 27th USENIX Security Symposium, pages 729--743, 2018.

[52]

Shankara Pailoor, Andrew Aday, and Suman Jana. MoonShine: optimizing OS fuzzer seed selection with trace distillation. In Proceedings of the 27th USENIX Security Symposium, pages 729--743, 2018.

[53]

Hui Peng and Mathias Payer. USBFuzz: A framework for fuzzing USB drivers by device emulation. In Proceedings of the 29th USENIX Security Symposium, pages 2559--2575, 2020.

[54]

Van-Thuan Pham, Marcel Böhme, and Abhik Roychoudhury. AFLNet: a greybox fuzzer for network protocols. In Proceedings of the 13th International Conference on Software Testing, Validation and Verification (ICST), pages 460--465, 2020.

[55]

Van-Thuan Pham, Marcel Böhme, Andrew E Santosa, Alexandru Răzvan Căciulescu, and Abhik Roychoudhury. Smart greybox fuzzing. IEEE Transactions on Software Engineering (TSE), 47(9):1980--1997, 2019.

[56]

Alexandre Rebert, Sang Kil Cha, Thanassis Avgerinos, Jonathan Foote, David Warren, Gustavo Grieco, and David Brumley. Optimizing seed selection for fuzzing. In Proceedings of the 23rd USENIX Security Symposium, pages 861--875, 2014.

[57]

RoboFuzz repository. https://github.com/sslab-gatech/RoboFuzz.

[58]

ROCK: the Robot COnstruction Kit. https://www.rock-robotics.org/.

[59]

ROS platform for building robot applications. https://www.ros.org/.

[60]

AMCL localization in ROS2 navigation. https://github.com/ros-planning/navigation2/tree/main/nav2_amcl.

[61]

Bt navigator in ROS2 navigation. https://github.com/ros-planning/navigation2/tree/main/nav2_bt_navigator.

[62]

Controller in ROS2 navigation. https://github.com/ros-planning/navigation2/tree/main/nav2_controller.

[63]

ROS2 Turtlebot3 map explorer. https://github.com/DaniGarciaLopez/ros2_explorer.

[64]

Ros2-fuzz: automatic fuzzing for ROS2. https://github.com/rosin-project/ros2_fuzz.

[65]

Ros2 navigation framework. https://github.com/ros-planning/navigation2.

[66]

Planner in ROS2 navigation. https://github.com/ros-planning/navigation2/tree/main/nav2_planner.

[67]

Recoveries in ROS2 navigation. https://github.com/ros-planning/navigation2/tree/main/nav2_recoveries.

[68]

Robots using ROS. https://robots.ros.org/.

[69]

2022 ROS2 metrics report. https://www.therobotreport.com/2022-ros-2-metrics-report/.

[70]

The rosbag package in ROS. http://wiki.ros.org/rosbag.

[71]

ROZZ website. https://sites.google.com/view/rozz-fuzzing/.

[72]

RTAB-Map: an application of real-time appearance-based mapping. https://github.com/introlab/rtabmap_ros.

[73]

André Santos, Alcino Cunha, and Nuno Macedo. Static-time extraction and analysis of the ROS computation graph. In Proceedings of the 2019 international conference on robotic computing (IRC), pages 62--69, 2019.

[74]

Sergej Schumilo, Cornelius Aschermann, Andrea Jemmett, Ali Abbasi, and Thorsten Holz. Nyx-net: network fuzzing with incremental snapshots. In Proceedings of the 17th European Conference on Computer Systems, pages 166--180, 2022.

[75]

Slam Toolbox: a set of tools and capabilities for 2D SLAM. https://github.com/SteveMacenski/slam_toolbox.

[76]

Prashast Srivastava and Mathias Payer. Gramatron: effective grammar-aware fuzzing. In Proceedings of the 2021 International Symposium on Software Testing and Analysis (ISSTA), pages 244--256, 2021.

[77]

Syzkaller: a kernel fuzzer. https://github.com/google/syzkaller.

[78]

Daimeng Wang, Zheng Zhang, Hang Zhang, Zhiyun Qian, Srikanth V Krishnamurthy, and Nael Abu-Ghazaleh. SyzVegas: beating kernel fuzzing odds with reinforcement learning. In Proceedings of the 30th USENIX Security Symposium, pages 2741--2758, 2021.

[79]

Trey Woodlief, Sebastian Elbaum, and Kevin Sullivan. Fuzzing mobile robot environments for fast automated crash detection. In Proceedings of the 2021 International Conference on Robotics and Automation (ICRA), pages 5417--5423, 2021.

Digital Library

[80]

Kai-Tao Xie, Jia-Ju Bai, Yong-Hao Zou, and Yu-Ping Wang. ROZZ: property-based fuzzing for robotic programs in ROS. In Proceedings of the 2022 International Conference on Robotics and Automation (ICRA), pages 6786--6792, 2022.

Digital Library

[81]

YARP: Yet Another Robot Platform. https://github.com/robotology/yarp.

[82]

Safdar Zaman, Gerald Steinbauer, Johannes Maurer, Peter Lepej, and Suzana Uran. An integrated model-based diagnosis and repair architecture for ros-based robot systems. In Proceedings of the 2013 International Conference on Robotics and Automation (ICRA), pages 482--489, 2013.

[83]

Rui Zhong, Yongheng Chen, Hong Hu, Hangfan Zhang, Wenke Lee, and Dinghao Wu. SQUIRREL: testing database management systems with language validity and coverage feedback. In Proceedings of the 27th International Conference on Computer and Communications Security (CCS), pages 955--970, 2020.

Digital Library

[84]

Yong-Hao Zou, Jia-Ju Bai, Jielong Zhou, Jianfeng Tan, Chenggang Qin, and Shi-Min Hu. TCP-Fuzz: detecting memory and semantic bugs in TCP stacks with fuzzing. In Proceedings of the 2021 USENIX Annual Technical Conference, pages 489--502, 2021.

Cited By

Caldas RGarcía JSchiopu MPelliccione PRodrigues GBerger T(2024)Runtime Verification and Field-Based Testing for ROS-Based Robotic SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.344469750:10(2544-2567)Online publication date: Oct-2024
https://doi.org/10.1109/TSE.2024.3444697
Wang YYang CHan RLi XMa J(2024)Dvatar: Simulating the Binary Firmware of DronesIEEE Internet of Things Journal10.1109/JIOT.2024.341644911:19(30661-30675)Online publication date: 1-Oct-2024
https://doi.org/10.1109/JIOT.2024.3416449

Recommendations

RoboFuzz: fuzzing robotic systems over robot operating system (ROS) for finding correctness bugs
ESEC/FSE 2022: Proceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering

Robotic systems are becoming an integral part of human lives. Responding to the increased demands for robot productions, Robot Operating System (ROS), an open-source middleware suite for robotic development, is gaining traction by providing ...
ROZZ: Property-based Fuzzing for Robotic Programs in ROS
2022 International Conference on Robotics and Automation (ICRA)
ROS is popular in robotic-software development, and thus detecting bugs in ROS programs is important for modern robots. Fuzzing is a promising technique of runtime testing. But existing fuzzing approaches are limited in testing ROS programs, due to ...
An architecture for multi-robot localization and mapping in the Gazebo/Robot Operating System simulation environment

Robots are an important part of urban search and rescue tasks. World wide attention has been given to developing capable physical platforms that would be beneficial for rescue teams. It is evident that use of multi-robots increases the effectiveness of ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ASPLOS '24: Proceedings of the 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2

April 2024

1299 pages

ISBN:9798400703850

DOI:10.1145/3620665

General Chairs:
Nael Abu-Ghazaleh,
Rajiv Gupta,
Program Chairs:
Madan Musuvathi,
Dan Tsafrir

Copyright © 2024 Copyright is held by the owner/author(s). Publication rights licensed to ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

In-Cooperation

SIGBED: ACM Special Interest Group on Embedded Systems

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 27 April 2024

Check for updates

Qualifiers

Research-article

Funding Sources

National Natural Science Foundation of China

Conference

ASPLOS '24

Sponsor:

ASPLOS '24: 29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2

April 27 - May 1, 2024

CA, La Jolla, USA

Acceptance Rates

Overall Acceptance Rate 535 of 2,713 submissions, 20%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
368
Total Downloads

Downloads (Last 12 months)368
Downloads (Last 6 weeks)34

Reflects downloads up to 11 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Caldas RGarcía JSchiopu MPelliccione PRodrigues GBerger T(2024)Runtime Verification and Field-Based Testing for ROS-Based Robotic SystemsIEEE Transactions on Software Engineering10.1109/TSE.2024.344469750:10(2544-2567)Online publication date: Oct-2024
https://doi.org/10.1109/TSE.2024.3444697
Wang YYang CHan RLi XMa J(2024)Dvatar: Simulating the Binary Firmware of DronesIEEE Internet of Things Journal10.1109/JIOT.2024.341644911:19(30661-30675)Online publication date: 1-Oct-2024
https://doi.org/10.1109/JIOT.2024.3416449

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Figures

Tables

Media

View Table of Conten