research-article

Open access

Parallelism-preserving automatic differentiation for second-order array languages

Authors:

Matthew J. Johnson,

Dougal MaclaurinAuthors Info & Claims

FHPNC 2021: Proceedings of the 9th ACM SIGPLAN International Workshop on Functional High-Performance and Numerical Computing

Pages 13 - 23

https://doi.org/10.1145/3471873.3472975

Published: 22 August 2021 Publication History

Abstract

We develop automatic differentiation (AD) procedures for reductions and scans—parameterized by arbitrary differentiable monoids—in a way that preserves parallelism, by rewriting them as other reductions and scans. This is in contrast with the literature and with existing AD systems, which are either general, but force sequential execution of the derivative program, or only include hand-crafted rules for a select few monoids (usually (0, +), (1, ×), (−∞, max) and (∞, min)) and thus lack the general flexibility of second-order languages.

References

[1]

[n.d.]. XLA: Optimizing Compiler for Machine Learning. https://www.tensorflow.org/xla Online; accessed 1 May 2021.

[2]

Martín Abadi, Paul Barham, Jianmin Chen, Zhifeng Chen, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Geoffrey Irving, Michael Isard, Manjunath Kudlur, Josh Levenberg, Rajat Monga, Sherry Moore, Derek G. Murray, Benoit Steiner, Paul Tucker, Vijay Vasudevan, Pete Warden, Martin Wicke, Yuan Yu, and Xiaoqiang Zheng. 2016. TensorFlow: A System for Large-Scale Machine Learning. In Proceedings of the 12th USENIX Conference on Operating Systems Design and Implementation (OSDI’16). USENIX Association, USA. 265–283. isbn:9781931971331

Digital Library

[3]

Martín Abadi and Gordon D. Plotkin. 2019. A Simple Differentiable Programming Language. Proc. ACM Program. Lang., 4, POPL (2019), Article 38, Dec., 28 pages. https://doi.org/10.1145/3371106

Digital Library

[4]

Gilbert Bernstein, Michael Mara, Tzu-Mao Li, Dougal Maclaurin, and Jonathan Ragan-Kelley. 2020. Differentiating A Tensor Language. arXiv preprint arXiv:2008.11256.

[5]

Guy E. Blelloch. 1993. Prefix Sums and Their Applications. In Synthesis of Parallel Algorithms, John H. Reif (Ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 35–60. isbn:978-1-55860-135-2

Digital Library

[6]

Guy E. Blelloch and Bruce M. Maggs. 2010. Parallel Algorithms (2 ed.). Chapman & Hall/CRC, 25. isbn:9781584888208

Digital Library

[7]

James Bradbury, Roy Frostig, Peter Hawkins, Matthew James Johnson, Chris Leary, Dougal Maclaurin, George Necula, Adam Paszke, Jake VanderPlas, Skye Wanderman-Milne, and Qiao Zhang. 2018. JAX: composable transformations of Python+NumPy programs. http://github.com/google/jax

[8]

Aloïs Brunel, Damiano Mazza, and Michele Pagani. 2019. Backpropagation in the Simply Typed Lambda-Calculus with Linear Negation. Proc. ACM Program. Lang., 4, POPL (2019), Article 64, Dec., 27 pages. https://doi.org/10.1145/3371132

Digital Library

[9]

Conal Elliott. 2018. The Simple Essence of Automatic Differentiation. Proc. ACM Program. Lang., 2, ICFP (2018), Article 70, July, 29 pages. https://doi.org/10.1145/3236765

Digital Library

[10]

Roy Frostig, Matthew Johnson, and Chris Leary. 2018. Compiling machine learning programs via high-level tracing. Machine Learning and Systems (MLSys), https://mlsys.org/Conferences/doc/2018/146.pdf

[11]

Roy Frostig, Matthew J. Johnson, Dougal Maclaurin, Adam Paszke, and Alexey Radul. 2021. Decomposing reverse-mode automatic differentiation. In LAFI 2021 workshop at POPL. arxiv:2105.09469

[12]

Troels Henriksen, Niels G. W. Serup, Martin Elsman, Fritz Henglein, and Cosmin E. Oancea. 2017. Futhark: Purely Functional GPU-Programming with Nested Parallelism and in-Place Array Updates. In Proceedings of the 38th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI 2017). Association for Computing Machinery, New York, NY, USA. 556–571. isbn:9781450349888 https://doi.org/10.1145/3062341.3062354

Digital Library

[13]

Anders Kiel Hovgaard, Troels Henriksen, and Martin Elsman. 2019. High-Performance Defunctionalisation in Futhark. In Trends in Functional Programming, Michał Pał ka and Magnus Myreen (Eds.). Springer International Publishing, Cham. 136–156. isbn:978-3-030-18506-0 https://doi.org/10.1007/978-3-030-18506-0_7

[14]

P. Hovland and C. Bischof. 1998. Automatic Differentiation for Message-Passing Parallel Programs. In Proceedings of the 12th. International Parallel Processing Symposium on International Parallel Processing Symposium (IPPS ’98). IEEE Computer Society, USA. 98–104. https://doi.org/10.1109/IPPS.1998.669896

Digital Library

[15]

Dougal Maclaurin, Alexey Radul, Matthew J. Johnson, and Dimitrios Vytiniotis. 2019. Dex: array programming with typed indices. NeurIPS workshop: Program Transformations for Machine Learning.

[16]

Kiminori Matsuzaki and Kento Emoto. 2010. Implementing Fusion-Equipped Parallel Skeletons by Expression Templates. In Implementation and Application of Functional Languages, Marco T. Morazán and Sven-Bodo Scholz (Eds.). Springer Berlin Heidelberg, Berlin, Heidelberg. 72–89. isbn:978-3-642-16478-1 https://doi.org/10.1007/978-3-642-16478-1_5

Digital Library

[17]

Damiano Mazza and Michele Pagani. 2021. Automatic Differentiation in PCF. Proc. ACM Program. Lang., 5, POPL (2021), Article 28, Jan., 27 pages. https://doi.org/10.1145/3434309

Digital Library

[18]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32, H. Wallach, H. Larochelle, A. Beygelzimer, F. d' Alché-Buc, E. Fox, and R. Garnett (Eds.). Curran Associates, Inc., 8024–8035. http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf

Digital Library

[19]

Adam Paszke, Daniel Johnson, David Duvenaud, Dimitrios Vytiniotis, Alexey Radul, Matthew Johnson, Jonathan Ragan-Kelley, and Dougal Maclaurin. 2021. Getting to the Point: Index Sets and Parallelism-Preserving Autodiff for Pointful Array Programming. Proc. ACM Program. Lang., 5, ICFP (2021), Article 88, Aug., https://doi.org/10.1145/3473593

Digital Library

[20]

John C. Reynolds. 1972. Definitional Interpreters for Higher-Order Programming Languages. In Proceedings of the ACM Annual Conference - Volume 2 (ACM ’72). Association for Computing Machinery, New York, NY, USA. 717–740. isbn:9781450374927 https://doi.org/10.1145/800194.805852

Digital Library

[21]

Justin Slepak, Olin Shivers, and Panagiotis Manolios. 2014. An Array-Oriented Language with Static Rank Polymorphism. In Programming Languages and Systems, Zhong Shao (Ed.). Springer Berlin Heidelberg, Berlin, Heidelberg. 27–46. isbn:978-3-642-54833-8 https://doi.org/10.1007/978-3-642-54833-8_3

Digital Library

[22]

Dimitrios Vytiniotis, Dan Belov, Richard Wei, Gordon Plotkin, and Martin Abadi. 2019. The differentiable curry. In NeurIPS 2019 Workshop Program Transformations. https://openreview.net/forum?id=ryxuz9SzDB

[23]

Fei Wang, Daniel Zheng, James Decker, Xilun Wu, Grégory M. Essertel, and Tiark Rompf. 2019. Demystifying Differentiable Programming: Shift/Reset the Penultimate Backpropagator. Proc. ACM Program. Lang., 3, ICFP (2019), Article 96, July, 31 pages. https://doi.org/10.1145/3341700

Digital Library

[24]

Yuan Yu, Martín Abadi, Paul Barham, Eugene Brevdo, Mike Burrows, Andy Davis, Jeff Dean, Sanjay Ghemawat, Tim Harley, Peter Hawkins, Michael Isard, Manjunath Kudlur, Rajat Monga, Derek Murray, and Xiaoqiang Zheng. 2018. Dynamic Control Flow in Large-Scale Machine Learning. In Proceedings of the Thirteenth EuroSys Conference (EuroSys ’18). Association for Computing Machinery, New York, NY, USA. Article 18, 15 pages. isbn:9781450355841 https://doi.org/10.1145/3190508.3190551

Digital Library

Cited By

Smeding TVákár M(2024)Efficient CHADProceedings of the ACM on Programming Languages10.1145/36328788:POPL(1060-1088)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632878
Schenck RRønning OHenriksen TOancea CWolf FShende SCulhane CAlam SJagode H(2022)AD for an array language with nested parallelismProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/3571885.3571962(1-15)Online publication date: 13-Nov-2022
https://dl.acm.org/doi/10.5555/3571885.3571962
Elsman MHenglein FKaarsgaard RMathiesen MSchenck R(2022)Combinatory Adjoints and DifferentiationElectronic Proceedings in Theoretical Computer Science10.4204/EPTCS.360.1360(1-26)Online publication date: 30-Jun-2022
https://doi.org/10.4204/EPTCS.360.1
Show More Cited By

Index Terms

Parallelism-preserving automatic differentiation for second-order array languages
1. Mathematics of computing
  1. Mathematical analysis
    1. Numerical analysis
      1. Automatic differentiation
2. Software and its engineering
  1. Software notations and tools
    1. General programming languages
      1. Language types
        Parallel programming languages

Recommendations

Critical multipliers in variational systems via second-order generalized differentiation

In this paper we introduce the notions of critical and noncritical multipliers for variational systems and extend to a general framework the corresponding notions by Izmailov and Solodov developed for classical Karush---Kuhn---Tucker (KKT) systems. It ...
A second-order optimality condition with first- and second-order complementarity associated with global convergence of algorithms

We develop a new notion of second-order complementarity with respect to the tangent subspace related to second-order necessary optimality conditions by the introduction of so-called tangent multipliers. We prove that around a local minimizer, a second-...
Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning
Abstract
Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

FHPNC 2021: Proceedings of the 9th ACM SIGPLAN International Workshop on Functional High-Performance and Numerical Computing

August 2021

49 pages

ISBN:9781450386142

DOI:10.1145/3471873

General Chairs:
Gabriele Keller
Utrecht University, Netherlands
,
Troels Henriksen
University of Copenhagen, Denmark

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Sponsors

SIGPLAN: ACM Special Interest Group on Programming Languages

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 22 August 2021

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

ICFP '21

Sponsor:

SIGPLAN

ICFP '21: 26th ACM SIGPLAN International Conference on Functional Programming

August 22, 2021

Virtual, Republic of Korea

Acceptance Rates

Overall Acceptance Rate 18 of 25 submissions, 72%

Upcoming Conference

ICFP '25

Sponsor:
sigplan

ACM SIGPLAN International Conference on Functional Programming

October 12 - 18, 2025

Singapore , Singapore

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
1,562
Total Downloads

Downloads (Last 12 months)228
Downloads (Last 6 weeks)21

Reflects downloads up to 15 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Smeding TVákár M(2024)Efficient CHADProceedings of the ACM on Programming Languages10.1145/36328788:POPL(1060-1088)Online publication date: 5-Jan-2024
https://dl.acm.org/doi/10.1145/3632878
Schenck RRønning OHenriksen TOancea CWolf FShende SCulhane CAlam SJagode H(2022)AD for an array language with nested parallelismProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/3571885.3571962(1-15)Online publication date: 13-Nov-2022
https://dl.acm.org/doi/10.5555/3571885.3571962
Elsman MHenglein FKaarsgaard RMathiesen MSchenck R(2022)Combinatory Adjoints and DifferentiationElectronic Proceedings in Theoretical Computer Science10.4204/EPTCS.360.1360(1-26)Online publication date: 30-Jun-2022
https://doi.org/10.4204/EPTCS.360.1
Schenck RRϕnning OHenriksen TOancea C(2022)AD for an Array Language with Nested ParallelismSC22: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41404.2022.00063(1-15)Online publication date: Nov-2022
https://doi.org/10.1109/SC41404.2022.00063

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents