research-article

Beautiful differentiation

Author:

Conal M. ElliottAuthors Info & Claims

ICFP '09: Proceedings of the 14th ACM SIGPLAN international conference on Functional programming

Pages 191 - 202

https://doi.org/10.1145/1596550.1596579

Published: 31 August 2009 Publication History

Abstract

Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The higher-dimensional case has also been tackled, though with extra complexity. This paper develops an implementation of higher-dimensional, higher-order, forward-mode AD in the extremely general and elegant setting of calculus on manifolds and derives that implementation from a simple and precise specification. In order to motivate and discover the implementation, the paper poses the question "What does AD mean, independently of implementation?" An answer arises in the form of naturality of sampling a function and its derivative. Automatic differentiation flows out of this naturality condition, together with the chain rule. Graduating from first-order to higher-order AD corresponds to sampling all derivatives instead of just one. Next, the setting is expanded to arbitrary vector spaces, in which derivative values are linear maps. The specification of AD adapts to this elegant and very general setting, which even simplifies the development.

Supplementary Material

JPG File (beautifuldierentiationonvimeo.jpg)

Download
15.22 KB

MP4 File (beautifuldierentiationonvimeo.mp4)

Download
126.54 MB

References

[1]

Manuel M. T. Chakravarty, Gabriele Keller, and Simon Peyton-Jones. Associated type synonyms. In ICFP '05: Proceedings of the tenth ACM SIGPLAN international conference on Functional programming, pages 241--253. ACM Press, 2005.

Digital Library

[2]

Conal Elliott. Beautiful differentiation (extended version). Technical Report 2009-02, LambdaPix, March 2009. URL http://conal.net/papers/beautiful-differentiation.

[3]

Conal Elliott. Denotational design with type class morphisms. Technical Report 2009-01, LambdaPix, March 2009. URL http://conal.net/papers/type-class-morphisms.

[4]

Conal Elliott. Push-pull functional reactive programming. In Proceedings of the Haskell Symposium, 2009.

Digital Library

[5]

Jason Foutz. Higher order multivariate automatic differentiation in haskell. Blog post, February 2008. URL http://metavar.blogspot.com/2008/02/higher-order- multivariate-automatic.html.

[6]

Ralf Hinze. Generalizing generalized tries. Journal of Functional Programming, 10 (04): 327--351, 2000.

Digital Library

[7]

Graham Hutton and Jeremy Gibbons. The generic approximation lemma. Information Processing Letters, 79 (4): 197--201, 2001.

Digital Library

[8]

Jerzy Karczmarczuk. Functional coding of differential forms. In Scottish Workshop on Functional Programming, 1999.

[9]

Jerzy Karczmarczuk. Functional differentiation of computer programs. Higher-Order and Symbolic Computation, 14 (1), 2001.

Digital Library

[10]

Conor McBride and Ross Paterson. Applicative programming with effects. Journal of Functional Programming, 18 (1): 1--13, 2008.

Digital Library

[11]

M. Douglas McIlroy. The music of streams. Information Processing Letters, 77 (2-4): 189--195, 2001.

Digital Library

[12]

Henrik Nilsson. Functional automatic differentiation with Dirac impulses. In Proceedings of the Eighth ACM SIGPLAN International Conference on Functional Programming, pages 153--164, Uppsala, Sweden, August 2003. ACM Press.

Digital Library

[13]

Barak A. Pearlmutter and Jeffrey Mark Siskind. Lazy multivariate higher-order forward-mode AD. In Proceedings of the 2007 Symposium on Principles of Programming Languages, pages 155--60, Nice, France, January 2007.

Digital Library

[14]

Simon Peyton Jones, Andrew Tolmach, and Tony Hoare. Playing by the rules: rewriting as a practical optimisation technique in ghc. In In Haskell Workshop. ACM SIGPLAN, 2001.

[15]

Jeffrey Mark Siskind and Barak A. Pearlmutter. Nesting forward-mode AD in a functional framework. Higher Order Symbolic Computation, 21 (4): 361--376, 2008.

Digital Library

[16]

Jeffrey Mark Siskind and Barak A. Pearlmutter. Perturbation confusion and referential transparency: Correct functional implementation of forward-mode AD. In Implementation and Application of Functional Languages, IFL'05, September 2005.

[17]

Michael Spivak. Calculus on Manifolds: A Modern Approach to Classical Theorems of Advanced Calculus. HarperCollins Publishers, 1971.

Cited By

van den Berg BSchrijvers TMcKinna JVandenbroucke A(2024)Forward- or reverse-mode automatic differentiationScience of Computer Programming10.1016/j.scico.2023.103010231:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.scico.2023.103010
Eriksson OPalmkvist VBroman DDe Roover CRumpe BShaikhha A(2023)Partial Evaluation of Automatic Differentiation for Differential-Algebraic Equations SolversProceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3624007.3624054(57-71)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3624007.3624054
Radul APaszke AFrostig RJohnson MMaclaurin D(2023)You Only Linearize Once: Tangents Transpose to GradientsProceedings of the ACM on Programming Languages10.1145/35712367:POPL(1246-1274)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3571236
Show More Cited By

Index Terms

Beautiful differentiation
1. Mathematics of computing
  1. Mathematical analysis
    1. Numerical analysis
      1. Numerical differentiation
    2. Quadrature

Recommendations

Beautiful differentiation
ICFP '09

Automatic differentiation (AD) is a precise, efficient, and convenient method for computing derivatives of functions. Its forward-mode implementation can be quite simple even when extended to compute all of the higher-order derivatives as well. The ...
Automatic differentiation of explicit Runge-Kutta methods for optimal control

This paper considers the numerical solution of optimal control problems based on ODEs. We assume that an explicit Runge-Kutta method is applied to integrate the state equation in the context of a recursive discretization approach. To compute the ...
Numerical solution of a third-order nonlinear boundary-value problem by automatic differentiation

We develop a simple numerical method for obtaining Taylor series approximation to the solution of a nonlinear third-order boundary-value problem. We use recursive formulas derived from the governing differential equation itself to calculate exact values ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICFP '09: Proceedings of the 14th ACM SIGPLAN international conference on Functional programming

August 2009

364 pages

ISBN:9781605583327

DOI:10.1145/1596550

General Chair:
Graham Hutton
University of Nottingham
,
Program Chair:
Andrew Tolmach
Portland State University

ACM SIGPLAN Notices Volume 44, Issue 9
ICFP '09
September 2009
343 pages
ISSN:0362-1340
EISSN:1558-1160
DOI:10.1145/1631687
Issue’s Table of Contents

Copyright © 2009 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 31 August 2009

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICFP '09

Sponsor:

ICFP '09: ACM SIGPLAN International Conference on Functional Programming

August 31 - September 2, 2009

Edinburgh, Scotland

Acceptance Rates

Overall Acceptance Rate 333 of 1,064 submissions, 31%

Upcoming Conference

ICFP '25

Sponsor:
sigplan

ACM SIGPLAN International Conference on Functional Programming

October 12 - 18, 2025

Singapore , Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

21
Total Citations
View Citations
652
Total Downloads

Downloads (Last 12 months)14
Downloads (Last 6 weeks)1

Reflects downloads up to 25 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

van den Berg BSchrijvers TMcKinna JVandenbroucke A(2024)Forward- or reverse-mode automatic differentiationScience of Computer Programming10.1016/j.scico.2023.103010231:COnline publication date: 1-Jan-2024
https://dl.acm.org/doi/10.1016/j.scico.2023.103010
Eriksson OPalmkvist VBroman DDe Roover CRumpe BShaikhha A(2023)Partial Evaluation of Automatic Differentiation for Differential-Algebraic Equations SolversProceedings of the 22nd ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3624007.3624054(57-71)Online publication date: 22-Oct-2023
https://dl.acm.org/doi/10.1145/3624007.3624054
Radul APaszke AFrostig RJohnson MMaclaurin D(2023)You Only Linearize Once: Tangents Transpose to GradientsProceedings of the ACM on Programming Languages10.1145/35712367:POPL(1246-1274)Online publication date: 11-Jan-2023
https://dl.acm.org/doi/10.1145/3571236
Elsman MHenglein FKaarsgaard RMathiesen MSchenck R(2022)Combinatory Adjoints and DifferentiationElectronic Proceedings in Theoretical Computer Science10.4204/EPTCS.360.1360(1-26)Online publication date: 30-Jun-2022
https://doi.org/10.4204/EPTCS.360.1
Jiménez-Pastor AJacob JPogudin G(2022)Exact Linear Reduction for Rational Dynamical SystemsComputational Methods in Systems Biology10.1007/978-3-031-15034-0_10(198-216)Online publication date: 14-Sep-2022
https://dl.acm.org/doi/10.1007/978-3-031-15034-0_10
Ishii H(2021)Automatic Differentiation with Higher Infinitesimals, or Computational Smooth Infinitesimal Analysis in Weil AlgebraComputer Algebra in Scientific Computing10.1007/978-3-030-85165-1_11(174-191)Online publication date: 16-Aug-2021
https://doi.org/10.1007/978-3-030-85165-1_11
Zhu SHung SChakrabarti SWu XDonaldson ATorlak E(2020)On the principles of differentiable quantum programming languagesProceedings of the 41st ACM SIGPLAN Conference on Programming Language Design and Implementation10.1145/3385412.3386011(272-285)Online publication date: 11-Jun-2020
https://dl.acm.org/doi/10.1145/3385412.3386011
Deisenroth MFaisal AOng C(2020)Mathematics for Machine Learning10.1017/9781108679930Online publication date: 20-Feb-2020
https://doi.org/10.1017/9781108679930
Wang FZheng DDecker JWu XEssertel GRompf T(2019)Demystifying differentiable programming: shift/reset the penultimate backpropagatorProceedings of the ACM on Programming Languages10.1145/33417003:ICFP(1-31)Online publication date: 26-Jul-2019
https://dl.acm.org/doi/10.1145/3341700
Wang FDecker JWu XEssertel GRompf T(2018)Backpropagation with continuation callbacksProceedings of the 32nd International Conference on Neural Information Processing Systems10.5555/3327546.3327682(10201-10212)Online publication date: 3-Dec-2018
https://dl.acm.org/doi/10.5555/3327546.3327682
Show More Cited By

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents