research-article

Open access

A general construction for abstract interpretation of higher-order automatic differentiation

Authors:

Gagandeep Singh,

Sasa MisailovicAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 6, Issue OOPSLA2

Article No.: 161, Pages 1007 - 1035

https://doi.org/10.1145/3563324

Published: 31 October 2022 Publication History

Abstract

We present a novel, general construction to abstractly interpret higher-order automatic differentiation (AD). Our construction allows one to instantiate an abstract interpreter for computing derivatives up to a chosen order. Furthermore, since our construction reduces the problem of abstractly reasoning about derivatives to abstractly reasoning about real-valued straight-line programs, it can be instantiated with almost any numerical abstract domain, both relational and non-relational. We formally establish the soundness of this construction.

We implement our technique by instantiating our construction with both the non-relational interval domain and the relational zonotope domain to compute both first and higher-order derivatives. In the latter case, we are the first to apply a relational domain to automatic differentiation for abstracting higher-order derivatives, and hence we are also the first abstract interpretation work to track correlations across not only different variables, but different orders of derivatives.

We evaluate these instantiations on multiple case studies, namely robustly explaining a neural network and more precisely computing a neural network’s Lipschitz constant. For robust interpretation, first and second derivatives computed via zonotope AD are up to 4.76× and 6.98× more precise, respectively, compared to interval AD. For Lipschitz certification, we obtain bounds that are up to 11,850× more precise with zonotopes, compared to the state-of-the-art interval-based tool.

References

[1]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, and Michael Isard. 2016. TensorFlow: A System for Large-Scale Machine Learning. In 12th USENIX symposium on operating systems design and implementation.

[2]

Aws Albarghouthi. 2021. Introduction to Neural Network Verification. Foundations and Trends® in Programming Languages.

[3]

David Alvarez-Melis and Tommi S Jaakkola. 2018. Towards robust interpretability with self-explaining neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems.

Digital Library

[4]

Marco Ancona, Enea Ceolini, Cengiz Öztireli, and Markus Gross. 2018. Towards better understanding of gradient-based attribution methods for Deep Neural Networks. In 6th International Conference on Learning Representations (ICLR).

[5]

Claus Bendtsen and Ole Stauning. 1996. FADBAD, a flexible C++ package for automatic differentiation.

[6]

Jesse Bettencourt, Matthew J Johnson, and David Duvenaud. 2019. Taylor-mode automatic differentiation for higher-order derivatives in JAX.

[7]

Patrick Cousot and Radhia Cousot. 1977. Abstract interpretation: a unified lattice model for static analysis of programs by construction or approximation of fixpoints. In Proceedings of the 4th ACM SIGACT-SIGPLAN symposium on Principles of programming languages.

Digital Library

[8]

Patrick Cousot, Roberto Giacobazzi, and Francesco Ranzato. 2019. A^2I: abstract^2 interpretation. Proceedings of the ACM on Programming Languages.

[9]

Jens Deussen. 2021. Global Derivatives. Ph. D. Dissertation.

[10]

Fr. Faa Di Bruno. 1857. Note sur une nouvelle formule de calcul différentiel. Quarterly J. Pure Appl. Math.

[11]

Pietro Di Gianantonio and Abbas Edalat. 2013. A language for differentiable functions. In International Conference on Foundations of Software Science and Computational Structures.

Digital Library

[12]

Thomas Fel, Mélanie Ducoffe, David Vigouroux, Rémi Cadène, Mikael Capelle, Claire Nicodème, and Thomas Serre. 2022. Don’t Lie to Me! Robust and Efficient Explainability with Verified Perturbation Analysis. arXiv preprint arXiv:2202.07728.

[13]

Jeffrey Fike and Juan Alonso. 2011. The Development of Hyper-Dual Numbers for Exact Second-Derivative Calculations. AIAA.

[14]

Cormac Flanagan, Amr Sabry, Bruce F Duba, and Matthias Felleisen. 1993. The essence of compiling with continuations. In Proceedings of the ACM SIGPLAN 1993 conference on Programming language design and implementation.

Digital Library

[15]

Oleg Fryazinov, Alexander Pasko, and Peter Comninos. 2010. Technical Section: Fast Reliable Interrogation of Procedurally Defined Implicit Surfaces Using Extended Revised Affine Arithmetic. Comput. Graph., 34, 6 (2010).

[16]

Khalil Ghorbal, Eric Goubault, and Sylvie Putot. 2009. The zonotope abstract domain taylor1+. In International Conference on Computer Aided Verification.

Digital Library

[17]

Andreas Griewank, Jean Utke, and Andrea Walther. 2000. Evaluating higher derivative tensors by forward propagation of univariate Taylor series. Mathematics of computation.

[18]

Andreas Griewank and Andrea Walther. 2008. Evaluating derivatives: principles and techniques of algorithmic differentiation. SIAM.

[19]

Horace He. 2019. The State of Machine Learning Frameworks in 2019. https://thegradient.pub/state-of-ml-frameworks-2019-pytorch-dominates-research-tensorflow-dominates-industry/ The Gradient.

[20]

Yuanming Hu, Luke Anderson, Tzu-Mao Li, Qi Sun, Nathan Carr, Jonathan Ragan-Kelley, and Frédo Durand. 2020. DiffTaichi: Differentiable Programming for Physical Simulation. In International Conference on Learning Representations.

[21]

Jan Hückelheim, Ziqing Luo, Sri Hari Krishna Narayanan, Stephen Siegel, and Paul D Hovland. 2018. Verifying Properties of Differentiable Programs. In International Static Analysis Symposium.

[22]

Mathieu Huot, Sam Staton, and Matthijs Vákár. 2021. Higher Order Automatic Differentiation of Higher Order Functions. arXiv preprint arXiv:2101.06757.

[23]

Fabian Immler. 2018. A Verified ODE Solver and Smale’s 14th Problem. Ph. D. Dissertation. Technische Universität München.

[24]

Joseph D Janizek, Pascal Sturmfels, and Su-In Lee. 2021. Explaining explanations: Axiomatic feature interactions for deep networks. Journal of Machine Learning Research, 22 (2021).

[25]

Bertrand Jeannet and Antoine Miné. 2009. Apron: A library of numerical abstract domains for static analysis. In International Conference on Computer Aided Verification. 661–667.

Digital Library

[26]

Matt Jordan and Alex Dimakis. 2021. Provable Lipschitz certification for generative models. In International Conference on Machine Learning. 5118–5126.

[27]

Jerzy Karczmarczuk. 2001. Functional differentiation of computer programs. Higher-order and symbolic computation.

[28]

Faustyna Krawiec, Neel Krishnaswami, Simon Peyton Jones, Tom Ellis, Andrew Fitzgibbon, and R Eisenberg. 2022. Provably correct, asymptotically efficient, higher-order reverse-mode automatic differentiation. Proceedings of the ACM on Programming Languages.

Digital Library

[29]

Jacob Laurel, Rem Yang, Gagandeep Singh, and Sasa Misailovic. 2022. A Dual Number Abstraction for Static Analysis of Clarke Jacobians. Proceedings of the ACM on Programming Languages.

Digital Library

[30]

Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Appendix to A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://jsl1994.github.io/papers/OOPSLA2022_appendix.pdf

[31]

Jacob Laurel, Rem Yang, Shubham Ugare, Robert Nagel, Gagandeep Singh, and Sasa Misailovic. 2022. Artifact for A General Construction for Abstract Interpretation of Higher-Order Automatic Differentiation. https://doi.org/10.1145/3554329

Digital Library

[32]

Yann LeCun. 1998. The MNIST database of handwritten digits. http://yann. lecun. com/exdb/mnist/.

[33]

Tzu-Mao Li, Michaël Gharbi, Andrew Adams, Frédo Durand, and Jonathan Ragan-Kelley. 2018. Differentiable programming for image processing and deep learning in Halide. ACM Transactions on Graphics (TOG), 37, 4 (2018).

Digital Library

[34]

Ravi Mangal, Kartik Sarangmath, Aditya V Nori, and Alessandro Orso. 2020. Probabilistic Lipschitz Analysis of Neural Networks. In International Static Analysis Symposium.

[35]

Antoine Miné. 2006. Symbolic methods to enhance the precision of numerical abstract domains. In International Workshop on Verification, Model Checking, and Abstract Interpretation.

[36]

Adam Paszke, Daniel D Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew J Johnson, Jonathan Ragan-Kelley, and Dougal Maclaurin. 2021. Getting to the point: index sets and parallelism-preserving autodiff for pointful array programming. Proceedings of the ACM on Programming Languages.

Digital Library

[37]

Colin Paterson, Haoze Wu, John Grese, Radu Calinescu, Corina S Păsăreanu, and Clark Barrett. 2021. Deepcert: Verification of contextually relevant robustness for neural network image classifiers. In International Conference on Computer Safety, Reliability, and Security.

Digital Library

[38]

Barak A Pearlmutter and Jeffrey Mark Siskind. 2007. Lazy Multivariate Higher-Order Forward-Mode AD. In Symposium on Principles of Programming Languages.

[39]

Thomas Reps and Aditya Thakur. 2016. Automating abstract interpretation. In International Conference on Verification, Model Checking, and Abstract Interpretation.

Digital Library

[40]

Wonryong Ryou, Jiayu Chen, Mislav Balunovic, Gagandeep Singh, Andrei Dan, and Martin Vechev. 2021. Scalable polyhedral verification of recurrent neural networks. In International Conference on Computer Aided Verification.

Digital Library

[41]

Benjamin Sherman, Jesse Michel, and Michael Carbin. 2021. λ _S: Computable Semantics for Differentiable Programming with Higher-Order Functions and Datatypes. Proceedings of the ACM on Programming Languages.

[42]

Zhouxing Shi, Huan Zhang, Kai-Wei Chang, Minlie Huang, and Cho-Jui Hsieh. 2019. Robustness Verification for Transformers. In International Conference on Learning Representations.

[43]

Gagandeep Singh, Timon Gehr, Matthew Mirman, Markus Püschel, and Martin T Vechev. 2018. Fast and Effective Robustness Certification. NeurIPS.

[44]

Gagandeep Singh, Timon Gehr, Markus Püschel, and Martin Vechev. 2019. An abstract domain for certifying neural networks. Proceedings of the ACM on Programming Languages.

Digital Library

[45]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2017. Fast polyhedra abstract domain. In Proceedings of the 44th ACM SIGPLAN Symposium on Principles of Programming Languages.

Digital Library

[46]

Gagandeep Singh, Markus Püschel, and Martin Vechev. 2018. A practical construction for decomposing numerical abstract domains. Proceedings of the ACM on Programming Languages.

Digital Library

[47]

Matthew Sotoudeh and Aditya V Thakur. 2020. Abstract Neural Networks. In International Static Analysis Symposium.

[48]

Ole Stauning. 1997. Automatic validation of numerical solutions.

[49]

Jorge Stolfi and Luiz Henrique de Figueiredo. 2003. An introduction to affine arithmetic. Trends in Computational and Applied Mathematics, 4 (2003).

[50]

Vassilis Vassiliadis, Jan Riehme, Jens Deussen, Konstantinos Parasyris, Christos D Antonopoulos, Nikolaos Bellas, Spyros Lalis, and Uwe Naumann. 2016. Towards automatic significance analysis for approximate computing. In 2016 IEEE/ACM International Symposium on Code Generation and Optimization.

Digital Library

[51]

Andrea Walther and Andreas Griewank. 2012. Getting Started with ADOL-C. Combinatorial Scientific Computing.

[52]

Rem Yang, Jacob Laurel, Sasa Misailovic, and Gagandeep Singh. 2022. Provable Defense Against Geometric Transformations. arXiv preprint arXiv:2207.11177.

Cited By

Laurel JQian SSingh GMisailovic S(2023)Synthesizing Precise Static Analyzers for Automatic DifferentiationProceedings of the ACM on Programming Languages10.1145/36228677:OOPSLA2(1964-1992)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3622867
Ugare SBanerjee DMisailovic SSingh G(2023)Incremental Verification of Neural NetworksProceedings of the ACM on Programming Languages10.1145/35912997:PLDI(1920-1945)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591299

Index Terms

A general construction for abstract interpretation of higher-order automatic differentiation
1. Mathematics of computing
  1. Mathematical analysis
    1. Numerical analysis
      1. Automatic differentiation
2. Theory of computation
  1. Semantics and reasoning
    1. Program reasoning
      1. Abstraction
      2. Program analysis

Recommendations

A dual number abstraction for static analysis of Clarke Jacobians

We present a novel abstraction for bounding the Clarke Jacobian of a Lipschitz continuous, but not necessarily differentiable function over a local input region. To do so, we leverage a novel abstract domain built upon dual numbers, adapted to soundly ...
Abstract interpretation of resolution-based semantics

We extend the abstract interpretation point of view on context-free grammars by Cousot and Cousot to resolution-based logic programs and proof systems. Starting from a transition-based small-step operational semantics of Prolog programs (akin to the ...
Abstract Interpretation as Automated Deduction

Automata theory, algorithmic deduction and abstract interpretation provide the foundation behind three approaches to implementing program verifiers. This article is a first step towards a mathematical translation between these approaches. By extending ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 6, Issue OOPSLA2

October 2022

1932 pages

EISSN:2475-1421

DOI:10.1145/3554307

Editor:
Philip Wadler
University of Edinburgh, UK

Issue’s Table of Contents

Copyright © 2022 Owner/Author.

Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for third-party components of this work must be honored. For all other uses, contact the Owner/Author.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 October 2022

Published in PACMPL Volume 6, Issue OOPSLA2

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Badges

Author Tags

Qualifiers

Research-article

Funding Sources

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
510
Total Downloads

Downloads (Last 12 months)185
Downloads (Last 6 weeks)18

Reflects downloads up to 25 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Laurel JQian SSingh GMisailovic S(2023)Synthesizing Precise Static Analyzers for Automatic DifferentiationProceedings of the ACM on Programming Languages10.1145/36228677:OOPSLA2(1964-1992)Online publication date: 16-Oct-2023
https://dl.acm.org/doi/10.1145/3622867
Ugare SBanerjee DMisailovic SSingh G(2023)Incremental Verification of Neural NetworksProceedings of the ACM on Programming Languages10.1145/35912997:PLDI(1920-1945)Online publication date: 6-Jun-2023
https://dl.acm.org/doi/10.1145/3591299

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Figures

Tables

Media

View Issue’s Table of Contents