research-article

Open access

Toward verified artificial intelligence

Authors:

Sanjit A. Seshia,

Dorsa Sadigh,

S. Shankar SastryAuthors Info & Claims

Communications of the ACM, Volume 65, Issue 7

Pages 46 - 55

https://doi.org/10.1145/3503914

Published: 21 June 2022 Publication History

All formats PDF

Abstract

Making AI more trustworthy with a formal methods-based approach to AI system verification and validation.

References

[1]

Censi, A., Slutsky, K., Wongpiromsarn, T., Yershov, D., Pendleton, S., Fu, J., and Frazzoli, E. Liability, ethics, and culture-aware behavior specification using rulebooks. In 2019 Intern. Conf. on Robotics and Automation (ICRA), IEEE, 8536--8542.

Google Scholar

[2]

Cohen, J.M., Rosenfeld, E., and Kolter, J.Z. Certified adversarial robustness via randomized smoothing. In Proceedings of the 36^th Intern. Conf. on Machine Learning 97 (2019), 1310--1320.

Google Scholar

[3]

Desai, A., Ghosh, S., Seshia, S.A., Shankar, N., and Tiwari, A. SOTER: A runtime assurance framework for programming safe robotics systems. In IEEE/IFIP Intern. Conf. on Dependable Systems and Networks (2019), 138--150.

Crossref

Google Scholar

[4]

Dietterich, T.G. and Horvitz, E.J. Rise of concerns about AI: Reflections and directions. Communications of the ACM 58, 10 (2015), 38--40.

Digital Library

Google Scholar

[5]

Dreossi, T., Donze, A., and Seshia, S.A. Compositional falsification of cyber-physical systems with machine learning components. In Proceedings of the NASA Formal Methods Conf. (Lecture Notes in Computer Science) 10227 (2017), 357--372.

Google Scholar

[6]

Dreossi, T., Fremont, D.J., Ghosh, S., Kim, E., Ravanbakhsh, H., Vazquez-Chanlatte, M., and Seshia, S.A. VerifAI: A toolkit for the formal design and analysis of artificial intelligence-based systems. In 31^st Intern. Conf. on Computer Aided Verification (Lecture Notes in Computer Science) 11561 (2017), 432--442.

Google Scholar

[7]

Dreossi, T., Jha, S., and Seshia, S.A. Semantic adversarial deep learning. In 30^th Intern. Conf. on Computer Aided Verification (Lecture Notes in Computer Science) 10981 (2018), 3--26.

Google Scholar

[8]

Fisac, J.F. et al. A general safety framework for learning-based control in uncertain robotic systems. IEEE Transactions on Automatic Control 64, 7 (2018), 2737--2752.

Google Scholar

[9]

Fremont, D.J., Donzé, A., Seshia, S.A., and Wessel, D. Control improvisation. In 35^th IARCS Annual Conf. on Foundations of Software Technology and Theoretical Computer Science (2015), 463--474.

Google Scholar

[10]

Fremont, D.J., Dreossi, T., Ghosh, S., Yue, X., Sangiovanni-Vincentelli, A.L., and Seshia, S.A. Scenic: A language for scenario specification and scene generation. In Proceedings of the 40^th annual ACM SIGPLAN Conf. on Programming Language Design and Implementation (2019).

Digital Library

Google Scholar

[11]

Garcia, J. and Fernández, F. A comprehensive survey on safe reinforcement learning. J. of Machine Learning Research 16, 1 (2015), 1437--1480.

Google Scholar

[12]

Gehr, T., Mirman, M., Drachsler-Cohen, D., Tsankov, P., Chaudhuri, S., and Vechev, M. AI2: Safety and robustness certification of neural networks with abstract interpretation. In IEEE Symposium on Security and Privacy (2018), 3--18.

Crossref

Google Scholar

[13]

Goodfellow, I., McDaniel, P., and Papernot, N. Making machine learning robust against adversarial inputs. Communications of the ACM 61, 7 (2018), 56--66.

Digital Library

Google Scholar

[14]

Huang, X., Kwiatkowska, M., Wang, S., and Wu, M. Safety verification of deep neural networks. In Intern. Conf. on Computer Aided Verification, Springer, (2017), 3--29.

Crossref

Google Scholar

[15]

Icarte, R.T., Klassen, T., Valenzano, R., and McIlraith, S. Using reward machines for high-level task specification and decomposition in reinforcement learning. Intern. Conf. on Machine Learning (2018), 2107--2116.

Google Scholar

[16]

Jha, S., Sahai, T., Raman, V., Pinto, A., and Francis, M. Explaining AI decisions using efficient methods for learning sparse Boolean formulae. J. of Automated Reasoning 63, 4 (2019), 1055--1075.

Digital Library

Google Scholar

[17]

Li, W., Sadigh, D., Sastry, S.S., and Seshia, S.A. Synthesis for human-in-the-loop control systems. In Proceedings of the 20^th Intern. Conf. on Tools and Algorithms for the Construction and Analysis of Systems (2014), 470--484.

Crossref

Google Scholar

[18]

Liu, C. et al. Algorithms for verifying deep neural networks. Foundations and Trends in Optimization 4, 3--4 (2021), 244--404.

Digital Library

Google Scholar

[19]

Meel, K.S. et al. Constrained sampling and counting: Universal hashing meets SAT solving. In Beyond NP, Papers from the 2016 AAAI Workshop, (Feb. 12, 2016).

Google Scholar

[20]

Milch, B., Marthi, B., Russell, S., Sontag, D., Ong, D.L., and Kolobov, A. BLOG: Probabilistic models with unknown objects. Statistical Relational Learning (2007), 373.

Google Scholar

[21]

Mitchell, T.M., Keller, R.M., and Kedar-Cabelli, S.T. Explanation-based generalization: A unifying view. Machine Learning 1, 1 (1986), 47--80.

Crossref

Google Scholar

[22]

Nghiem, T. et al. Monte-Carlo techniques for falsification of temporal properties of non-linear hybrid systems. In Proceedings of the 13^th ACM Intern. Conf. on Hybrid Systems: Computation and Control (2010), 211--220.

Digital Library

Google Scholar

[23]

Nuzzo, P., Li, J., Sangiovanni-Vincentelli, A.L., Xi, Y., and Li, D. Stochastic assume-guarantee contracts for cyber-physical system design. ACM Transactions on Embedded Computing Systems 18, 1, (Jan. 2019).

Digital Library

Google Scholar

[24]

Pearl, J. The seven tools of causal inference, with reflections on machine learning. Communications of the ACM 62, 3 (2019), 54--60.

Digital Library

Google Scholar

[25]

Puggelli, A., Li, W., Sangiovanni-Vincentelli, A.L., and Seshia, S.A. Polynomial-time verification of PCTL properties of MDPs with convex uncertainties. In Proceedings of the 25^th Intern. Conf. on Computer-Aided Verification (2013).

Crossref

Google Scholar

[26]

Russell, S.J. and Norvig, P. Artificial Intelligence: A Modern Approach. Prentice Hall (2010).

Digital Library

Google Scholar

[27]

Sadigh, D. et al. Data-driven probabilistic modeling and verification of human driver behavior. In Formal Verification and Modeling in Human-Machine Systems, AAAI Spring Symposium (2014).

Google Scholar

[28]

Sadigh, D., Sastry, S., Seshia, S.A., and Dragan, A.D. Information gathering actions over human internal state. In Proceedings of the IEEE/RSJ Intern. Conf. on Intelligent Robots and Systems (2016).

Digital Library

Google Scholar

[29]

Selsam, D., Liang, P., and Dill, D.L. Developing bug-free machine learning systems with formal mathematics. In Proceedings of the 34^th Intern. Conf. on Machine Learning 70 (2017), 3047--3056.

Digital Library

Google Scholar

[30]

Seshia, S.A. Combining induction, deduction, and structure for verification and synthesis. In Proceedings of the IEEE 103, 11 (2015), 2036--2051.

Crossref

Google Scholar

[31]

Seshia, S.A. Introspective environment modeling. 19^th Intern. Conf. on Runtime Verification (2019), 15--26.

Digital Library

Google Scholar

[32]

Seshia, S.A. et al. Formal specification for deep neural networks. In Proceedings of the Intern. Symp. on Automated Technology for Verification and Analysis (2018), 20--34.

Crossref

Google Scholar

[33]

Sha, L. Using simplicity to control complexity. IEEE Software 18, 4 (2001), 20--28.

Google Scholar

[34]

Shoukry, Y., et al. SMC: Satisfiability modulo convex optimization. In Proceedings of the 10^th Intern. Conf. on Hybrid Systems: Computation and Control (2017).

Google Scholar

[35]

Tomlin, C., Mitchell, I., Bayen, A.M., and Oishi, M. Computational techniques for the verification of hybrid systems. In Proceedings of the IEEE 91, 7 (2003), 986--1001.

Crossref

Google Scholar

[36]

Tran, H-D., Xiang, W., and Johnson, T.T. Verification approaches for learning-enabled autonomous cyber-physical systems. IEEE Design & Test (Aug. 2020).

Crossref

Google Scholar

[37]

Vazquez-Chanlatte, M., Jha, S., Tiwari, A., Ho, M.K., and Seshia, S.A. Learning task Specifications from demonstrations. In Advances in Neural Information Processing Systems 31 (2018), 5372--5382.

Google Scholar

[38]

Wing, J.M. A specifier's introduction to formal methods. IEEE Computer 23, 9 (Sept. 1990), 8--24.

Digital Library

Google Scholar

[39]

Wing, J.M. Trustworthy AI. Communications of the ACM 64, 10 (2021), 64--71.

Digital Library

Google Scholar

[40]

Xu, J., Zhang, Z., Friedman, T., Liang, Y., and Van den Broeck, G. A semantic loss function for deep learning with symbolic knowledge. In Proceedings of the 35^th Intern. Conf. on Machine Learning (2018), 5498--5507.

Google Scholar

Cited By

View all

Belardinelli FJamroga WMittelmann MMurano ADastani MSichman JAlechina NDignum V(2024)Verification of Stochastic Multi-Agent Systems with Forgetful StrategiesProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3662863(160-169)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3662863
Paardekooper JBorth M(2024)Toward a Methodology for the Verification and Validation of AI-Based SystemsSAE International Journal of Connected and Automated Vehicles10.4271/12-08-01-00068:1Online publication date: 27-Jul-2024
https://doi.org/10.4271/12-08-01-0006
Bergami G(2024)DECLAREd: A Polytime LTLf FragmentLogics10.3390/logics20200042:2(79-111)Online publication date: 31-May-2024
https://doi.org/10.3390/logics2020004
Show More Cited By

Index Terms

Toward verified artificial intelligence

Recommendations

Artificial intelligence
Abstract
Artificial intelligence (AI) is the Science and Engineering domain concerned with the theory and practice of developing systems that exhibit the characteristics we associate with intelligence in human behavior. Starting with a brief history of ...
Artificial Intelligence: Theories, Models and Applications 7th Hellenic Conference on AI, SETN 2012, Lamia, Greece, May 28-31, 2012, Proceedings
ARTIFICIAL INTELLIGENCE LAW

Comments

Information & Contributors

Information

Published In

Communications of the ACM Volume 65, Issue 7

July 2022

85 pages

ISSN:0001-0782

EISSN:1557-7317

DOI:10.1145/3544941

Editor:
James Larus
Association for Computing Machinery, New York, NY

Issue’s Table of Contents

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 June 2022

Published in CACM Volume 65, Issue 7

Check for updates

Qualifiers

Research-article
Popular
Refereed

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

69
Total Citations
View Citations
22,318
Total Downloads

Downloads (Last 12 months)5,230
Downloads (Last 6 weeks)266

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

View all

Belardinelli FJamroga WMittelmann MMurano ADastani MSichman JAlechina NDignum V(2024)Verification of Stochastic Multi-Agent Systems with Forgetful StrategiesProceedings of the 23rd International Conference on Autonomous Agents and Multiagent Systems10.5555/3635637.3662863(160-169)Online publication date: 6-May-2024
https://dl.acm.org/doi/10.5555/3635637.3662863
Paardekooper JBorth M(2024)Toward a Methodology for the Verification and Validation of AI-Based SystemsSAE International Journal of Connected and Automated Vehicles10.4271/12-08-01-00068:1Online publication date: 27-Jul-2024
https://doi.org/10.4271/12-08-01-0006
Bergami G(2024)DECLAREd: A Polytime LTLf FragmentLogics10.3390/logics20200042:2(79-111)Online publication date: 31-May-2024
https://doi.org/10.3390/logics2020004
Bergami G(2024)Streamlining Temporal Formal Verification over Columnar DatabasesInformation10.3390/info1501003415:1(34)Online publication date: 8-Jan-2024
https://doi.org/10.3390/info15010034
Yang SSun BGuang HWang RZheng BGong WWang YShi YTong ZZhang LYang KLiu XCao YLi JWang Y(2024)CHAINS: CHAIN-Based Fusion Safety System Framework for Intelligent Connected VehicleCHAIN10.23919/CHAIN.2024.0000061:1(2-45)Online publication date: Mar-2024
https://doi.org/10.23919/CHAIN.2024.000006
Saha SSandha SAggarwal MWang BHan LBriseno JSrivastava M(2024)TinyNS: Platform-aware Neurosymbolic Auto Tiny Machine LearningACM Transactions on Embedded Computing Systems10.1145/360317123:3(1-48)Online publication date: 11-May-2024
https://dl.acm.org/doi/10.1145/3603171
Oliveira YSobrinho ÁSilva LSantos DGorgônio KPerkusich A(2024)Coloured Petri Nets Modeling Multilayer Perceptron Neural Networks2024 IEEE International Conference on Consumer Electronics (ICCE)10.1109/ICCE59016.2024.10444319(1-4)Online publication date: 6-Jan-2024
https://doi.org/10.1109/ICCE59016.2024.10444319
Liu HFeng S(2024)Curse of rarity for autonomous vehiclesNature Communications10.1038/s41467-024-49194-015:1Online publication date: 5-Jun-2024
https://doi.org/10.1038/s41467-024-49194-0
Kubyshkina EPrimiero G(2024)A possible worlds semantics for trustworthy non-deterministic computationsInternational Journal of Approximate Reasoning10.1016/j.ijar.2024.109212172(109212)Online publication date: Sep-2024
https://doi.org/10.1016/j.ijar.2024.109212
Schmies LHemmleb MBettge D(2024)Relevant input data for crack feature segmentation with deep learning on SEM imagery and topography dataEngineering Failure Analysis10.1016/j.engfailanal.2023.107814156(107814)Online publication date: Feb-2024
https://doi.org/10.1016/j.engfailanal.2023.107814
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Digital Edition

View this article in digital edition.

Digital Edition

Magazine Site

View this article on the magazine site (external)

Magazine Site

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Abstract

References

Cited By

Index Terms

Recommendations

Artificial intelligence

Artificial Intelligence: Theories, Models and Applications 7th Hellenic Conference on AI, SETN 2012, Lamia, Greece, May 28-31, 2012, Proceedings

ARTIFICIAL INTELLIGENCE LAW

Comments

Information

Published In

Publisher

Publication History

Check for updates

Qualifiers

Funding Sources

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

View options

PDF

eReader

Digital Edition

Magazine Site

Get Access

Login options

Full Access

Figures

Other

Share

Share this Publication link

Share on social media

Affiliations