BERN-NN: Tight Bound Propagation For Neural Networks Using Bernstein Polynomial Interval Arithmetic
Article No.: 19, Pages 1 - 11
Abstract
In this paper, we present BERN-NN as an efficient tool to perform bound propagation of Neural Networks (NNs). Bound propagation is a critical step in wide range of NN model checkers and reachability analysis tools. Given a bounded input set, bound propagation algorithms aim to compute tight bounds on the output of the NN. So far, linear and convex optimizations have been used to perform bound propagation. Since neural networks are highly non-convex, state-of-the-art bound propagation techniques suffer from introducing large errors. To circumvent such drawback, BERN-NN approximates the bounds of each neuron using a class of polynomials called Bernstein polynomials. Bernstein polynomials enjoy several interesting properties that allow BERN-NN to obtain tighter bounds compared to those relying on linear and convex approximations. BERN-NN is efficiently parallelized on graphic processing units (GPUs). Extensive numerical results show that bounds obtained by BERN-NN are orders of magnitude tighter than those obtained by state-of-the-art verifiers such as linear programming and linear interval arithmetic. Moreoveer, BERN-NN is both faster and produces tighter outputs compared to convex programming approaches like alpha-CROWN.
References
[1]
Ross Anderson, Joey Huchette, Will Ma, Christian Tjandraatmadja, and Juan Pablo Vielma. 2020. Strong mixed-integer programming formulations for trained neural networks. Mathematical Programming 183, 1 (2020), 3–39. https://doi.org/10.1007/s10107-020-01474-5
[2]
Stanley Bak, Changliu Liu, and Taylor T. Johnson. 2021. The Second International Verification of Neural Networks Competition (VNN-COMP 2021): Summary and Results. CoRR abs/2109.00498 (2021), 1–15.
[3]
Osbert Bastani, Yani Ioannou, Leonidas Lampropoulos, Dimitrios Vytiniotis, Aditya Nori, and Antonio Criminisi. 2016. Measuring Neural Net Robustness with Constraints. In Advances in Neural Information Processing Systems, Vol. 29. Association for Computing Machinery, Barcelona, Spain, 2613–2621.
[4]
Elena Botoeva, Panagiotis Kouvaros, Jan Kronqvist, Alessio Lomuscio, and Ruth Misener. 2020. Efficient verification of relu-based neural networks via dependency analysis. Proceedings of the AAAI Conference on Artificial Intelligence 34, 4 (2020), 3291–3299.
[5]
Rudy Bunel, Jingyue Lu, Ilker Turkaslan, P Kohli, P Torr, and P Mudigonda. 2020. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research 21, 42 (2020), 1–39.
[6]
Chih-Hong Cheng, Georg Nührenberg, and Harald Ruess. 2017. Maximum Resilience of Artificial Neural Networks. In Automated Technology for Verification and Analysis, Deepak D’Souza and K. Narayan Kumar (Eds.). Springer, 251–268. https://doi.org/10.1007/978-3-319-68167-2_18
[7]
Louis De Branges. 1959. The stone-weierstrass theorem. Proc. Amer. Math. Soc. 10, 5 (1959), 822–824.
[8]
Souradeep Dutta, Xin Chen, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. 2019. Sherlock-a tool for verification of neural network feedback systems: demo abstract. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control. 262–263.
[9]
Souradeep Dutta, Xin Chen, and Sriram Sankaranarayanan. 2019. Reachability analysis for neural feedback systems using regressive polynomial rule inference. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control. 157–168.
[10]
Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy A Mann, and Pushmeet Kohli. 2018. A Dual Approach to Scalable Verification of Deep Networks. In Uncertainty in Artificial Intelligence, Amir Globerson and Ricardo Silva (Eds.). Vol. 1. 550–559.
[11]
Jiameng Fan, Chao Huang, Xin Chen, Wenchao Li, and Qi Zhu. 2020. Reachnn*: A tool for reachability analysis of neural-network controlled systems. In International Symposium on Automated Technology for Verification and Analysis. Springer, 537–542.
[12]
Rida T Farouki. 2012. The Bernstein polynomial basis: A centennial retrospective. Computer Aided Geometric Design 29, 6 (2012), 379–419.
[13]
Rida T Farouki and VT Rajan. 1988. Algorithms for polynomials in Bernstein form. Computer Aided Geometric Design 5, 1 (1988), 1–26.
[14]
Mahyar Fazlyab, Alexander Robey, Hamed Hassani, Manfred Morari, and George Pappas. 2019. Efficient and accurate estimation of lipschitz constants for deep neural networks. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.). Vol. 32. Curran Associates, Inc., 11423–11434.
[15]
Matteo Fischetti and Jason Jo. 2018. Deep neural networks and mixed integer linear optimization. Constraints 23, 3 (2018), 296–309. https://doi.org/10.1007/s10601-018-9285-6
[16]
Daniel J Fremont, Johnathan Chiu, Dragos D Margineantu, Denis Osipychev, and Sanjit A Seshia. 2020. Formal analysis and redesign of a neural network-based aircraft taxiing system with VerifAI. In International Conference on Computer Aided Verification. Springer, 122–134.
[17]
Jürgen Garloff. 1985. Convergent bounds for the range of multivariate polynomials. In International Symposium on Interval Mathematics. Springer, 37–56.
[18]
Jürgen Garloff and Andrew P Smith. 2007. Guaranteed affine lower bound functions for multivariate polynomials. In PAMM: Proceedings in Applied Mathematics and Mechanics, Vol. 7. Wiley Online Library, 1022905–1022906.
[19]
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. AI2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). IEEE, 3–18. https://doi.org/10.1109/SP.2018.00058
[20]
Patrick Henriksen and Alessio Lomuscio. 2021. DEEPSPLIT: An efficient splitting method for neural network verification via indirect effect analysis. In IJCAI. 2549–2555.
[21]
Chao Huang, Jiameng Fan, Xin Chen, Wenchao Li, and Qi Zhu. 2022. Polar: A polynomial arithmetic framework for verifying neural-network controlled systems. In International Symposium on Automated Technology for Verification and Analysis. Springer, 414–430.
[22]
Radoslav Ivanov, James Weimer, Rajeev Alur, George J Pappas, and Insup Lee. 2019. Verisig: verifying safety properties of hybrid systems with neural network controllers. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control(HSCC ’19). Association for Computing Machinery, New York, NY, USA, 169–178. https://doi.org/10.1145/3302504.3311806
[23]
Guy Katz, Clark Barrett, David L. Dill, Kyle Julian, and Mykel J. Kochenderfer. 2017. Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In Computer Aided Verification (Cham, 2017) (Lecture Notes in Computer Science), Rupak Majumdar and Viktor Kunčak (Eds.). Springer International Publishing, 97–117. https://doi.org/10.1007/978-3-319-63387-9_5
[24]
Haitham Khedr, James Ferlez, and Yasser Shoukry. 2021. PEREGRiNN: Penalized-Relaxation Greedy Neural Network Verifier. In Computer Aided Verification, Alexandra Silva and K. Rustan M. Leino (Eds.). Springer International Publishing, Cham, 287–300.
[25]
Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, Mykel J Kochenderfer, 2021. Algorithms for verifying deep neural networks. Foundations and Trends® in Optimization 4, 3-4 (2021), 244–404.
[26]
Diana Andreea Popescu and Rogelio Tomas Garcia. 2016. Multivariate polynomial multiplication on GPU. Procedia Computer Science 80 (2016), 154–165.
[27]
Shashwati Ray and PSV Nataraj. 2012. A Matrix Method for Efficient Computation of Bernstein Coefficients.Reliab. Comput. 17, 1 (2012), 40–71.
[28]
Javier Sánchez-Reyes. 2003. Algebraic manipulation in the Bernstein form made simple via convolutions. Computer-Aided Design 35, 10 (2003), 959–967.
[29]
Andrew Paul Smith. 2009. Fast construction of constant bound functions for sparse polynomials. Journal of Global Optimization 43, 2 (2009), 445–458.
[30]
Volker Stahl. 1995. Interval methods for bounding the range of polynomials and solving systems of nonlinear equations. Ph. D. Dissertation. Johannes Kepler University Linz.
[31]
Xiaowu Sun, Haitham Khedr, and Yasser Shoukry. 2019. Formal verification of neural network controlled autonomous systems. In Proceedings of the 22nd ACM International Conference on Hybrid Systems: Computation and Control. 147–156.
[32]
Xiaowu Sun and Yasser Shoukry. 2022. Neurosymbolic motion and task planning for linear temporal logic tasks. arXiv preprint arXiv:2210.05180 (2022).
[33]
Vincent Tjeng, Kai Y. Xiao, and Russ Tedrake. 2019. Evaluating Robustness of Neural Networks with Mixed Integer Programming. In International Conference on Learning Representations.
[34]
Hoang-Dung Tran, Xiaodong Yang, Diego Manzanas Lopez, Patrick Musau, Luan Viet Nguyen, Weiming Xiang, Stanley Bak, and Taylor T. Johnson. 2020. NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systems. In Computer Aided Verification, Shuvendu K. Lahiri and Chao Wang (Eds.). Springer International Publishing, 3–17. https://doi.org/10.1007/978-3-030-53288-8_1
[35]
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Efficient formal safety analysis of neural networks. In Advances in Neural Information Processing Systems, S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (Eds.). Vol. 31. 6367–6377.
[36]
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In Proceedings of the 27th USENIX Conference on Security Symposium(SEC’18). USENIX Association, 1599–1614. https://doi.org/10.5555/3277203.3277323
[37]
Shiqi Wang, Huan Zhang, Kaidi Xu, Xue Lin, Suman Jana, Cho-Jui Hsieh, and J Zico Kolter. 2021. Beta-CROWN: Efficient Bound Propagation with Per-neuron Split Constraints for Neural Network Robustness Verification. In Advances in Neural Information Processing Systems.
[38]
Eric Wong and Zico Kolter. 2018. Provable defenses against adversarial examples via the convex outer adversarial polytope. In International conference on machine learning. 5286–5295.
[39]
Weiming Xiang, Hoang-Dung Tran, and Taylor T Johnson. 2018. Output reachable set estimation and verification for multilayer neural networks. IEEE transactions on neural networks and learning systems 29, 11 (2018), 5777–5783. https://doi.org/10.1109/TNNLS.2018.2808470
[40]
Weiming Xiang, Hoang-Dung Tran, Joel A. Rosenfeld, and Taylor T. Johnson. 2018. Reachable Set Estimation and Safety Verification for Piecewise Linear Systems with Neural Network Controllers. In 2018 Annual American Control Conference (ACC). 1574–1579.
[41]
Kaidi Xu, Huan Zhang, Shiqi Wang, Yihan Wang, Suman Jana, Xue Lin, and Cho-Jui Hsieh. 2021. Fast and Complete: Enabling Complete Neural Network Verification with Rapid and Massively Parallel Incomplete Verifiers. In ICLR.
Recommendations
Nearly-tight VC-dimension and pseudodimension bounds for piecewise linear neural networks
We prove new upper and lower bounds on the VC-dimension of deep neural networks with the ReLU activation function. These bounds are tight for almost the entire range of parameters. Letting W be the number of weights and L be the number of layers, we ...
Sharp upper and lower bounds for the moments of Bernstein polynomials
We give upper and lower bounds for the moments and the uniform moments of Bernstein polynomials. Asymptotically, such bounds are best possible.
Stability and stabilization of polynomial dynamical systems using Bernstein polynomials
HSCC '15: Proceedings of the 18th International Conference on Hybrid Systems: Computation and ControlIn this work, we examine relaxations for the stability analysis and synthesis of stabilizing controllers for polynomial dynamical systems. It is well-known that such problems can be naturally solved using a reduction to polynomial optimization problems. ...
Comments
Information & Contributors
Information
Published In
May 2023
239 pages
ISBN:9798400700330
DOI:10.1145/3575870
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 09 May 2023
Check for updates
Author Tags
Qualifiers
- Research-article
- Research
- Refereed limited
Conference
HSCC '23
Sponsor:
HSCC '23: 26th ACM International Conference on Hybrid Systems: Computation and Control
May 9 - 12, 2023
TX, San Antonio, USA
Acceptance Rates
Overall Acceptance Rate 153 of 373 submissions, 41%
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 383Total Downloads
- Downloads (Last 12 months)284
- Downloads (Last 6 weeks)35
Reflects downloads up to 22 Sep 2024
Other Metrics
Citations
View Options
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML FormatGet Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in