The Case for Scalable Quantitative Neural Network Analysis
Pages 22 - 25
Abstract
Neural networks are an increasingly common tool for solving problems that require complex analysis and pattern matching, such as identifying stop signs in a self driving car or processing medical imagery during diagnosis. Accordingly, verification of neural networks for safety and correctness is of great importance, as mispredictions can have catastrophic results in safety critical domains. As neural networks are known to be sensitive to small changes in input, leading to vulnerabilities and adversarial attacks, analyzing the robustness of networks to small changes in input is a key piece of evaluating their safety and correctness. However, there are many real-world scenarios where the requirements of robustness are not clear cut, and it is crucial to develop measures that assess the level of robustness of a given neural network model and compare levels of robustness across different models, rather than using a binary characterization such as robust vs. not robust.
We believe there is great need for developing scalable quantitative robustness verification techniques for neural networks. Formal verification techniques can provide guarantees of correctness, but most existing approaches do not provide quantitative robustness measures and are not effective in analyzing real-world network sizes. On the other hand, sampling-based quantitative robustness is not hindered much by the size of networks but cannot provide sound guarantees of quantitative results. We believe more research is needed to address the limitations of both symbolic and sampling-based verification approaches and create sound, scalable techniques for quantitative robustness verification of neural networks.
References
[1]
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. 2014. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on audio, speech, and language processing, 22, 10 (2014), 1533–1545.
[2]
Ron Amadeo. 2023. NYPD robocops: Hulking, 400-lb robots will start patrolling New York City. https://arstechnica.com/gadgets/2023/04/nypd-robocops-hulking-400-lb-robots-will-start-patrolling-new-york-city/
[3]
Teodora Baluta, Zheng Leong Chua, Kuldeep S Meel, and Prateek Saxena. 2021. Scalable quantitative verification for deep neural networks. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE). 312–323. https://doi.org/10.1109/icse43902.2021.00039
[4]
Teodora Baluta, Shiqi Shen, Shweta Shinde, Kuldeep S Meel, and Prateek Saxena. 2019. Quantitative verification of neural networks and its security applications. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security. 1249–1264. https://doi.org/10.1145/3319535.3354245
[5]
Mariusz Bojarski, Davide Del Testa, Daniel Dworakowski, Bernhard Firner, Beat Flepp, Prasoon Goyal, Lawrence D Jackel, Mathew Monfort, Urs Muller, and Jiakai Zhang. 2016. End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316.
[6]
Akhilan Boopathy, Tsui-Wei Weng, Pin-Yu Chen, Sijia Liu, and Luca Daniel. 2019. Cnn-cert: An efficient framework for certifying robustness of convolutional neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence. 33, 3240–3247. https://doi.org/10.1609/aaai.v33i01.33013240
[7]
Rudy Bunel, P Mudigonda, Ilker Turkaslan, P Torr, Jingyue Lu, and Pushmeet Kohli. 2020. Branch and bound for piecewise linear neural network verification. Journal of Machine Learning Research, 21, 2020 (2020), https://doi.org/10.23919/acc.2018.8431048
[8]
Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. 2020. nuscenes: A multimodal dataset for autonomous driving. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 11621–11631. https://doi.org/10.1109/cvpr42600.2020.01164
[9]
Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40, 4 (2017), 834–848. https://doi.org/10.1109/tpami.2017.2699184
[10]
Abené Clayton. 2023. Fire chief warns against ‘unleashing’ self-driving taxis in San Francisco. https://www.theguardian.com/us-news/2023/jun/23/self-driving-taxis-fire-chief-san-francisco
[11]
Nicols Cruz and Javier Ruiz-del Solar. 2020. Closing the simulation-to-reality gap using generative neural networks: Training object detectors for soccer robotics in simulation as a case study. In 2020 International Joint Conference on Neural Networks (IJCNN). 1–8. https://doi.org/10.1109/ijcnn48605.2020.9207173
[12]
Marko Dimjašević, Falk Howar, Kasper Luckow, and Zvonimir Rakamarić. 2018. Study of integrating random and symbolic testing for object-oriented software. In Integrated Formal Methods: 14th International Conference, IFM 2018, Maynooth, Ireland, September 5-7, 2018, Proceedings 14. 89–109. https://doi.org/10.1007/978-3-319-98938-9_6
[13]
Souradeep Dutta, Susmit Jha, Sriram Sankaranarayanan, and Ashish Tiwari. 2018. Output range analysis for deep feedforward neural networks. In NASA Formal Methods Symposium. 121–138. https://doi.org/10.1007/978-3-319-77935-5_9
[14]
Krishnamurthy Dvijotham, Robert Stanforth, Sven Gowal, Timothy A Mann, and Pushmeet Kohli. 2018. A Dual Approach to Scalable Verification of Deep Networks. In UAI. 1, 3.
[15]
Ruediger Ehlers. 2017. Formal verification of piece-wise linear feed-forward neural networks. In International Symposium on Automated Technology for Verification and Analysis. 269–286. https://doi.org/10.1007/978-3-319-68167-2_19
[16]
Timon Gehr, Matthew Mirman, Dana Drachsler-Cohen, Petar Tsankov, Swarat Chaudhuri, and Martin Vechev. 2018. Ai2: Safety and robustness certification of neural networks with abstract interpretation. In 2018 IEEE Symposium on Security and Privacy (SP). 3–18. https://doi.org/10.1109/sp.2018.00058
[17]
Divya Gopinath, Kaiyuan Wang, Mengshi Zhang, Corina S Pasareanu, and Sarfraz Khurshid. 2018. Symbolic execution for deep neural networks. arXiv preprint arXiv:1807.10439.
[18]
Alex Graves and Navdeep Jaitly. 2014. Towards end-to-end speech recognition with recurrent neural networks. In International conference on machine learning. 1764–1772.
[19]
Chris Isidore. 2021. Another Tesla reportedly using Autopilot hits a parked police car. https://www.cnn.com/2021/08/30/business/tesla-crash-police-car/index.html
[20]
Guy Katz, Clark Barrett, David L Dill, Kyle Julian, and Mykel J Kochenderfer. 2017. Reluplex: An efficient SMT solver for verifying deep neural networks. In International Conference on Computer Aided Verification. 97–117. https://doi.org/10.1007/978-3-319-63387-9_5
[21]
Guy Katz, Derek A Huang, Duligur Ibeling, Kyle Julian, Christopher Lazarus, Rachel Lim, Parth Shah, Shantanu Thakoor, Haoze Wu, and Aleksandar Zeljić. 2019. The marabou framework for verification and analysis of deep neural networks. In International Conference on Computer Aided Verification. 443–452. https://doi.org/10.1007/978-3-030-25540-4_26
[22]
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. 2017. Imagenet classification with deep convolutional neural networks. Commun. ACM, 60, 6 (2017), 84–90.
[23]
Yuan Y Liu, Min Yang, Malcolm Ramsay, Xiao S Li, and Jeremy W Coid. 2011. A comparison of logistic regression, classification and regression tree, and neural networks models in predicting violent re-offending. Journal of Quantitative Criminology, 27 (2011), 547–573. https://doi.org/10.1007/s10940-011-9137-7
[24]
Ravi Mangal, Aditya V Nori, and Alessandro Orso. 2019. Robustness of neural networks: A probabilistic and practical approach. In 2019 IEEE/ACM 41st International Conference on Software Engineering: New Ideas and Emerging Results (ICSE-NIER). 93–96. https://doi.org/10.1109/icse-nier.2019.00032
[25]
Erin Marquis. 2023. Whistleblower Drops 100 Gigabytes of Tesla Secrets to German News Site: Report. https://jalopnik.com/whistleblower-drops-100-gigabytes-of-tesla-secrets-to-g-1850476542
[26]
OpenAI. 2022. ChatGPT. https://chat.openai.com
[27]
Corina Păsăreanu, Hayes Converse, Antonio Filieri, and Divya Gopinath. 2020. On the probabilistic analysis of neural networks. In Proceedings of the IEEE/ACM 15th International Symposium on Software Engineering for Adaptive and Self-Managing Systems. 5–8.
[28]
Joseph Redmon, Santosh Divvala, Ross Girshick, and Ali Farhadi. 2016. You only look once: Unified, real-time object detection. In Proceedings of the IEEE conference on computer vision and pattern recognition. 779–788. https://doi.org/10.1109/cvpr.2016.91
[29]
Joshua Saxe and Konstantin Berlin. 2015. Deep neural network based malware detection using two dimensional binary program features. In 2015 10th international conference on malicious and unwanted software (MALWARE). 11–20. https://doi.org/10.1109/malware.2015.7413680
[30]
Dinggang Shen, Guorong Wu, and Heung-Il Suk. 2017. Deep learning in medical image analysis. Annual review of biomedical engineering, 19 (2017), 221–248.
[31]
Faiz Siddiqui and Jeremy B. Merrill. 2023. 17 fatalities, 736 crashes: The shocking toll of Tesla’s Autopilot. https://www.washingtonpost.com/technology/2023/06/10/tesla-autopilot-crashes-elon-musk/
[32]
Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2018. Boosting robustness certification of neural networks. In International Conference on Learning Representations.
[33]
Guardian Staff. 2023. US air force denies running simulation in which AI drone ‘killed’ operator. https://www.theguardian.com/us-news/2023/jun/01/us-military-drone-ai-killed-operator-simulated-test
[34]
Nick Stephens, John Grosen, Christopher Salls, Andrew Dutcher, Ruoyu Wang, Jacopo Corbetta, Yan Shoshitaishvili, Christopher Kruegel, and Giovanni Vigna. 2016. Driller: Augmenting Fuzzing Through Selective Symbolic Execution. https://doi.org/10.14722/ndss.2016.23368
[35]
Brad Templeton. 2023. Tesla Again Paints A Crash Data Story That Misleads Many Readers. https://www.forbes.com/sites/bradtempleton/2023/04/26/tesla-again-paints-a-very-misleading-story-with-their-crash-data/?sh=7c533ab8feda
[36]
Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. 2018. Deeptest: Automated testing of deep-neural-network-driven autonomous cars. In Proceedings of the 40th international conference on software engineering. 303–314.
[37]
Muhammad Usman, Divya Gopinath, and Corina S Păsăreanu. 2021. QuantifyML: How Good is my Machine Learning Model? arXiv preprint arXiv:2110.12588.
[38]
Pranshu Verma and Will Oremus. 2023. ChatGPT invented a sexual harassment scandal and named a real law prof as the accused. The Washington Post, Apr, https://www.washingtonpost.com/technology/2023/04/05/chatgpt-lies/
[39]
Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. 2018. Formal security analysis of neural networks using symbolic intervals. In 27th $USENIX$ Security Symposium ($USENIX$ Security 18). 1599–1614.
[40]
Stefan Webb, Tom Rainforth, Yee Whye Teh, and M Pawan Kumar. 2018. A statistical approach to assessing neural network robustness. arXiv preprint arXiv:1811.07209.
[41]
Huan Zhang, Tsui-Wei Weng, Pin-Yu Chen, Cho-Jui Hsieh, and Luca Daniel. 2018. Efficient neural network robustness certification with general activation functions. arXiv preprint arXiv:1811.00866.
[42]
Lei Zhao, Yue Duan, Heng Yin, and Jifeng Xuan. 2019. Send Hardest Problems My Way: Probabilistic Path Prioritization for Hybrid Fuzzing. In NDSS. https://doi.org/10.14722/ndss.2019.23504
Index Terms
- The Case for Scalable Quantitative Neural Network Analysis
Recommendations
Quantitative Robustness Analysis of Neural Networks
ISSTA 2023: Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and AnalysisNeural networks are an increasingly common tool for solving problems that require complex analysis and pattern matching, such as identifying stop signs or processing medical imagery. Accordingly, verification of neural networks for safety and ...
Quantitative Verification of Neural Networks and Its Security Applications
CCS '19: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications SecurityNeural networks are increasingly employed in safety-critical domains. This has prompted interest in verifying or certifying logically encoded properties of neural networks. Prior work has largely focused on checking existential properties, wherein the ...
Expediting Neural Network Verification via Network Reduction
ASE '23: Proceedings of the 38th IEEE/ACM International Conference on Automated Software EngineeringA wide range of verification methods have been proposed to verify the safety properties of deep neural networks ensuring that the networks function correctly in critical applications. However, many well-known verification tools still struggle with ...
Comments
Information & Contributors
Information
Published In
December 2023
25 pages
ISBN:9798400703799
DOI:10.1145/3617574
- General Chairs:
- Marsha Chechik,
- Sebastian Elbaum,
- Boyue Caroline Hu,
- Lina Marsso,
- Meriel von Stein
Copyright © 2023 Owner/Author.
This work is licensed under a Creative Commons Attribution 4.0 International License.
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 04 December 2023
Check for updates
Author Tags
Qualifiers
- Research-article
Funding Sources
Conference
SE4SafeML '23
Sponsor:
SE4SafeML '23: 1st International Workshop on Dependability and Trustworthiness of Safety-Critical Systems with Machine Learned Components
December 4, 2023
CA, San Francisco, USA
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 150Total Downloads
- Downloads (Last 12 months)108
- Downloads (Last 6 weeks)12
Reflects downloads up to 08 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in