Abstract
Verifying learning robotic systems is challenging. Existing techniques and tools for verification of an artificial neural network (ANN) are concerned with component-level properties. Here, we deal with robotic systems whose control software uses ANN components, and with properties of that software that may depend on all components. Our focus is on trained fully connected ReLU neural networks for control. We present an approach to (1) modelling ANN components as part of behavioural models for control software and (2) verification using traditional and ANN-specific verification tools. We describe our results in the context of RoboChart, a domain-specific modelling language for robotics with support for formal verification. We describe our modelling notation and a strategy for automated proof using Isabelle and Marabou, for example.
This is a preview of subscription content, log in via an institution to check access.
Access this chapter
Tax calculation will be finalised at checkout
Purchases are for personal use only
Similar content being viewed by others
Notes
- 1.
Available at robostar.cs.york.ac.uk/robotool/.
- 2.
All the artefacts related to this validation work are available at github.com/UoY-RoboStar/robochart-ann-components/.
References
Ahn, J.-H., Rhee, K., You, Y.: A study on the collision avoidance of a ship using neural networks and fuzzy logic. Appl. Ocean Res. 37, 162–173 (2012)
An, P.E., Harris, C.J., Tribe, R., Clarke, N.: Aspects of neural networks in intelligent collision avoidance systems for prometheus. In: Joint Framework for Information Technology, pp. 129–135 (1993)
Attala, Z.: Verification of RoboChart models with ANN components. Technical report, University of York (2023). https://robostar.cs.york.ac.uk/publications/reports/Ziggy_Attala_Draft_Thesis.pdf
Attala, Z., Cavalcanti, A.L.C., Woodcock, J.C.P.: A comparison of neural network tools for the verification of linear specifications of ReLU networks. In: Albarghouthi, A., Katz, G., Narodytska, N. (eds.) 3rd Workshop on Formal Methods for ML-Enabled Autonomous System, pp. 22–33 (2020)
Brucker, A.D., Stell, A.: Verifying feedforward neural networks for classification in Isabelle/HOL. In: Chechik, M., Katoen, J.P., Leucker, M. (eds.) FM 2023. LNCS, vol. 14000, pp. 427–444. Springer, Cham (2023). https://doi.org/10.1007/978-3-031-27481-7_24
Clavière, A., Asselin, E., Garion, C., Pagetti, C.: Safety verification of neural network controlled systems. CoRR, abs/2011.05174 (2020)
Dreossi, T., et al.: Counterexample-guided data augmentation. arXiv:1805.06962 (2018)
Dupont, G., Aït-Ameur, Y., Pantel, M., Singh, N.K.: Event-B refinement for continuous behaviours approximation. In: Hou, Z., Ganesh, V. (eds.) ATVA 2021. LNCS, vol. 12971, pp. 320–336. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88885-5_21
Foster, S., Baxter, J., Cavalcanti, A.L.C., Woodcock, J.C.P., Zeyda, F.: Unifying semantic foundations for automated verification tools in Isabelle/UTP. Sci. Comput. Program. 197, 102510 (2020)
Foster, S., et al.: Unifying theories of reactive design contracts. CoRR, abs/1712.10233 (2017)
Foster, S., et al.: Automated verification of reactive and concurrent programs by calculation. CoRR, abs/2007.13529 (2020)
Gibson-Robinson, T., Armstrong, P., Boulgakov, A., Roscoe, A.W.: FDR3—a modern refinement checker for CSP. In: Ábrahám, E., Havelund, K. (eds.) TACAS 2014. LNCS, vol. 8413, pp. 187–201. Springer, Heidelberg (2014). https://doi.org/10.1007/978-3-642-54862-8_13
Hoare, C.A.R., Jifeng, H.: Unifying Theories of Programming. Prentice-Hall, Englewood Cliff (1998)
Hoare, C.A.R.: Communicating Sequential Processes. Prentice Hall International, Englewood Cliff (1985)
Hodge, V.J., Hawkins, R., Alexander, R.: Deep reinforcement learning for drone navigation using sensor data. Neural Comput. Appl. 33, 2015–2033 (2020). https://doi.org/10.1007/s00521-020-05097-x
Hu, B.C., et al.: If a human can see it, so should your system. In: Proceedings of the 44th International Conference on Software Engineering. ACM (2022)
Jacoby, Y., Barrett, C.W., Katz, G.: Verifying recurrent neural networks using invariant inference. CoRR, abs/2004.02462 (2020)
Julian, K.D., Kochenderfer, M.J.: Guaranteeing safety for neural network-based aircraft collision avoidance systems. In: 2019 IEEE/AIAA 38th Digital Avionics Systems Conference (DASC). IEEE (2019)
Julian, K.D., Kochenderfer, M.J., Owen, M.P.: Deep neural network compression for aircraft collision avoidance systems. J. Guid. Control. Dyn. 42(3), 598–608 (2019)
Katz, G., Barrett, C., Dill, D.L., Julian, K., Kochenderfer, M.J.: Reluplex: an efficient SMT solver for verifying deep neural networks. In: Majumdar, R., Kunčak, V. (eds.) CAV 2017. LNCS, vol. 10426, pp. 97–117. Springer, Cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5
Katz, G., et al.: The marabou framework for verification and analysis of deep neural networks. In: Dillig, I., Tasiran, S. (eds.) CAV 2019. LNCS, vol. 11561, pp. 443–452. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-25540-4_26
LeCun, Y., Bengio, Y., Hinton, G.: Deep learning. Nature 521, 436–444 (2015). https://doi.org/10.1038/nature14539
Miyazawa, A., Cavalcanti, A.: Formal refinement in SysML. In: Albert, E., Sekerinski, E. (eds.) IFM 2014. LNCS, vol. 8739, pp. 155–170. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-10181-1_10
Miyazawa, A., Ribeiro, P., Li, W., Cavalcanti, A., Timmis, J., Woodcock, J.: RoboChart: modelling and verification of the functional behaviour of robotic applications. Softw. Syst. Model. 18, 3097–3149 (2019)
Neves, A.C., González, I., Leander, J., Karoumi, R.: A new approach to damage detection in bridges using machine learning. In: Conte, J.P., Astroza, R., Benzoni, G., Feltrin, G., Loh, K.J., Moaveni, B. (eds.) EVACES 2017. LNCE, vol. 5, pp. 73–84. Springer, Cham (2018). https://doi.org/10.1007/978-3-319-67443-8_5
Nordmann, A., Hochgeschwender, N., Wrede, S.: A survey on domain-specific languages in robotics. In: Brugali, D., Broenink, J.F., Kroeger, T., MacDonald, B.A. (eds.) SIMPAR 2014. LNCS (LNAI), vol. 8810, pp. 195–206. Springer, Cham (2014). https://doi.org/10.1007/978-3-319-11900-7_17
Nwankpa, C., et al.: Activation functions: comparison of trends in practice and research for deep learning. arXiv:1811.03378 (2018)
Austin, P.D., Welch, P.H.: CSP for JavaTM (JCSP) 1.1-RC4 API specification (2008). https://www.cs.kent.ac.uk/projects/ofa/jcsp/jcsp-1.1-rc4/jcsp-doc/
ProofPower-Z reference manual (2006)
Rojas, R.: Neural Networks – A Systematic Introduction, chap. 7. Springer, Heidelberg (1996). https://doi.org/10.1007/978-3-642-61068-4
Roscoe, A.W.: Understanding Concurrent Systems. Texts in Computer Science. Springer, London (2011). https://doi.org/10.1007/978-1-84882-258-0
Roscoe, A.W.: The Theory and Practice of Concurrency. Prentice-Hall, Englewood Cliff (1997)
Singh, G., Gehr, T., Mirman, M., Püschel, M., Vechev, M.: Fast and effective robustness certification. In: Bengio, S., Wallach, H., Larochelle, H., Grauman, K., Cesa-Bianchi, N., Garnett, R. (eds.) Advances in Neural Information Processing Systems, vol. 31, pp. 10802–10813. Curran Associates Inc. (2018)
Spivey, J.M.: The Z Notation: A Reference Manual. Prentice-Hall, Englewood Cliff (1992)
Tran, H.-D., et al.: Star-based reachability analysis of deep neural networks. In: ter Beek, M.H., McIver, A., Oliveira, J.N. (eds.) FM 2019. LNCS, vol. 11800, pp. 670–686. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-30942-8_39
University of Oxford. FDR Manual, May 2020. Release 4.2.7. dl.cocotec.io/fdr/fdr-manual.pdf. Accessed 31 May 2020
Woodcock, J., Davies, J.: Using Z. Prentice Hall, Englewood Cliff (1996)
Ye, K., Foster, S., Woodcock, J.: Automated reasoning for probabilistic sequential programs with theorem proving. In: Fahrenberg, U., Gehrke, M., Santocanale, L., Winter, M. (eds.) RAMiCS 2021. LNCS, vol. 13027, pp. 465–482. Springer, Cham (2021). https://doi.org/10.1007/978-3-030-88701-8_28
Acknowledgements
This work has been funded by the UK EPSRC Grants EP/R025479/1, and EP/V026801/2, and by the UK Royal Academy of Engineering Grant No CiET1718/45.
Author information
Authors and Affiliations
Corresponding author
Editor information
Editors and Affiliations
Copyright information
© 2023 The Author(s), under exclusive license to Springer Nature Switzerland AG
About this paper
Cite this paper
Attala, Z., Cavalcanti, A., Woodcock, J. (2023). Modelling and Verifying Robotic Software that Uses Neural Networks. In: Ábrahám, E., Dubslaff, C., Tarifa, S.L.T. (eds) Theoretical Aspects of Computing – ICTAC 2023. ICTAC 2023. Lecture Notes in Computer Science, vol 14446. Springer, Cham. https://doi.org/10.1007/978-3-031-47963-2_3
Download citation
DOI: https://doi.org/10.1007/978-3-031-47963-2_3
Published:
Publisher Name: Springer, Cham
Print ISBN: 978-3-031-47962-5
Online ISBN: 978-3-031-47963-2
eBook Packages: Computer ScienceComputer Science (R0)