article

Towards compositional and generative tensor optimizations

Authors:

Adilla Susungi,

Norman A. Rink,

Jerónimo Castrillón,

Claude Tadonki,

Jochen FröhlichAuthors Info & Claims

ACM SIGPLAN Notices, Volume 52, Issue 12

Pages 169 - 175

https://doi.org/10.1145/3170492.3136050

Published: 23 October 2017 Publication History

Abstract

Many numerical algorithms are naturally expressed as operations on tensors (i.e. multi-dimensional arrays). Hence, tensor expressions occur in a wide range of application domains, e.g. quantum chemistry and physics; big data analysis and machine learning; and computational fluid dynamics. Each domain, typically, has developed its own strategies for efficiently generating optimized code, supported by tools such as domain-specific languages, compilers, and libraries. However, strategies and tools are rarely portable between domains, and generic solutions typically act as ''black boxes'' that offer little control over code generation and optimization. As a consequence, there are application domains without adequate support for easily generating optimized code, e.g. computational fluid dynamics. In this paper we propose a generic and easily extensible intermediate language for expressing tensor computations and code transformations in a modular and generative fashion. Beyond being an intermediate language, our solution also offers meta-programming capabilities for experts in code optimization. While applications from the domain of computational fluid dynamics serve to illustrate our proposed solution, we believe that our general approach can help unify research in tensor optimizations and make solutions more portable between domains.

References

[1]

2017. NumPy, package for scientific computing with Python. http: //www.numpy.org/ . (2017).

[2]

2017. XLA: Accelerated Linear Algebra. https://www.tensorflow.org/ performance/xla/ . (2017).

[3]

2017. Xtensor, Multi-dimensional arrays with broadcasting and lazy computing. https://github.com/QuantStack/xtensor . (2017).

[4]

Martín Abadi and Ashish Agarwal et al. 2015. TensorFlow: LargeScale Machine Learning on Heterogeneous Distributed Systems. http://download.tensorflow.org/paper/whitepaper2015.pdf. (2015).

[5]

Lénaïc Bagnères, Oleksandr Zinenko, Stéphane Huot, and Cédric Bastoul. 2016. Opening Polyhedral Compiler’s Black Box. In Proceedings of the 2016 International Symposium on Code Generation and Optimization (CGO ’16). ACM, New York, NY, USA, 128–138.

Digital Library

[6]

G. Baumgartner, A. Auer, D. E. Bernholdt, A. Bibireata, V. Choppella, D. Cociorva, Xiaoyang Gao, R. J. Harrison, S. Hirata, S. Krishnamoorthy, S. Krishnan, Chi chung Lam, Qingda Lu, M. Nooijen, R. M. Pitzer, J. Ramanujam, P. Sadayappan, and A. Sibiryakov. 2005. Synthesis of High-Performance Parallel Programs for a Class of ab Initio Quantum Chemistry Models. Proc. IEEE 93, 2 (Feb 2005), 276–292.

[7]

James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David WardeFarley, and Yoshua Bengio. 2010. Theano: a CPU and GPU Math Expression Compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy).

[8]

Chun Chen, Jacqueline Chame, and Mary Hall. 2008. CHiLL: A framework for composing high-level loop transformations. Technical Report. Technical Report 08-897, University of Southern California.

[9]

Albert Cohen, SÃľbastien Donadio, Maria-Jesus Garzaran, Christoph Herrmann, Oleg Kiselyov, and David Padua. 2006. In search of a program generator to implement generic transformations for highperformance computing. Science of Computer Programming 62, 1 (2006), 25–46.

Digital Library

[10]

Albert Cohen, Sylvain Girbal, and Olivier Temam. 2004. A Polyhedral Approach to Ease the Composition of Program Transformations. Springer Berlin Heidelberg, Berlin, Heidelberg, 292–303.

[11]

Albert Cohen, Marc Sigler, Sylvain Girbal, Olivier Temam, David Parello, and Nicolas Vasilache. 2005. Facilitating the Search for Compositions of Program Transformations. In Proceedings of the 19th Annual International Conference on Supercomputing (ICS ’05). ACM, New York, NY, USA, 151–160.

Digital Library

[12]

M. O. Deville, P. F. Fischer, and E. H. Mund. 2002. High-Order Methods for Incompressible Fluid Flow. Cambridge University Press, Cambridge.

[13]

Immo Huismann, Jörg Stiller, and Jochen Fröhlich. 2016. Fast Static Condensation for the Helmholtz Equation in a Spectral-Element Discretization. Springer International Publishing, Cham, 371–380.

[14]

Immo Huismann, Jörg Stiller, and Jochen Fröhlich. 2017. Factorizing the factorization — a spectral-element solver for elliptic equations with linear operation count. J. Comput. Phys. 346 (2017), 437–448.

[15]

Khaled Z. Ibrahim, Samuel W. Williams, Evgeny Epifanovsky, and Anna I. Krylov. 2014. Analysis and tuning of libtensor framework on multicore architectures. In 21st International Conference on High Performance Computing, HiPC 2014, Goa, India, December 17-20, 2014. 1–10.

[16]

Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman Amarasinghe. 2017. The Tensor Algebra Compiler. Technical Report. Massachusetts Institute of Technology.

[17]

P. M. W. Knijnenburg, T. Kisuki, and M. F. P. O’Boyle. 2002. Embedded Processor Design Challenges. Springer-Verlag New York, Inc., New York, NY, USA, Chapter Iterative Compilation, 171–187. http://dl.acm. org/citation.cfm?id=765198.765209

[18]

M. Puschel, J. M. F. Moura, J. R. Johnson, D. Padua, M. M. Veloso, B. W. Singer, Jianxin Xiong, F. Franchetti, A. Gacic, Y. Voronenko, K. Chen, R. W. Johnson, and N. Rizzolo. 2005. SPIRAL: Code Generation for DSP Transforms. Proc. IEEE 93, 2 (Feb 2005), 232–275.

[19]

Zu-Qing Qu. 2004. Static Condensation. Springer London, London, 47–70.

[20]

Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman Amarasinghe. 2013. Halide: A Language and Compiler for Optimizing Parallelism, Locality, and Recomputation in Image Processing Pipelines. In Proceedings of the 34th ACM SIGPLAN Conference on Programming Language Design and Implementation (PLDI ’13). ACM, New York, NY, USA, 519–530.

Digital Library

[21]

Florian Rathgeber, David A. Ham, Lawrence Mitchell, Michael Lange, Fabio Luporini, Andrew T. T. Mcrae, Gheorghe-Teodor Bercea, Graham R. Markall, and Paul H. J. Kelly. 2016. Firedrake: Automating the Finite Element Method by Composing Abstractions. ACM Trans. Math. Softw. 43, 3, Article 24 (Dec. 2016), 27 pages.

Digital Library

[22]

Gabe Rudy, Malik Murtaza Khan, Mary Hall, Chun Chen, and Jacqueline Chame. 2011. A Programming Language Interface to Describe Transformations and Code Generation. Springer Berlin Heidelberg, Berlin, Heidelberg, 136–150.

[23]

Paul Springer, Tong Su, and Paolo Bientinesi. 2017. HPTT: A Highperformance Tensor Transposition C++ Library. In Proceedings of the 4th ACM SIGPLAN International Workshop on Libraries, Languages, and Compilers for Array Programming (ARRAY 2017). ACM, New York, NY, USA, 56–62.

Digital Library

[24]

Michel Steuwer, Toomas Remmelg, and Christophe Dubach. 2017. Lift: A Functional Data-parallel IR for High-performance GPU Code Generation. In Proceedings of the 2017 International Symposium on Code Generation and Optimization (CGO ’17). IEEE Press, Piscataway, NJ, USA, 74–85. http://dl.acm.org/citation.cfm?id=3049832.3049841

[25]

Adilla Susungi, Albert Cohen, and Claude Tadonki. 2017. More Data Locality for Static Control Programs on NUMA Architectures. In Proceedings of the 7th International Workshop on Polyhedral Compilation Techniques (IMPACT ’17).

Cited By

Susungi ATadonki C(2021)Intermediate Representations for Explicitly Parallel ProgramsACM Computing Surveys10.1145/345229954:5(1-24)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1145/3452299
Susungi ATadonki C(2021)Intermediate Representations for Explicitly Parallel ProgramsACM Computing Surveys10.1145/345229954:5(1-24)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1145/3452299
Susungi ARink NCohen ACastrillon JTadonki C(2020)Meta-programming for cross-domain tensor optimizationsACM SIGPLAN Notices10.1145/3393934.327813153:9(79-92)Online publication date: 7-Apr-2020
https://dl.acm.org/doi/10.1145/3393934.3278131
Show More Cited By

Index Terms

Towards compositional and generative tensor optimizations
1. Software and its engineering
  1. Software notations and tools

Recommendations

CFDlang: High-level code generation for high-order methods in fluid dynamics
RWDSL2018: Proceedings of the Real World Domain Specific Languages Workshop 2018

Numerical simulations continue to enable fast and enormous progress in science and engineering. Writing efficient numerical codes is a difficult challenge that encompasses a variety of tasks from designing the right algorithms to exploiting the full ...
Towards compositional and generative tensor optimizations
GPCE 2017: Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences

Many numerical algorithms are naturally expressed as operations on tensors (i.e. multi-dimensional arrays). Hence, tensor expressions occur in a wide range of application domains, e.g. quantum chemistry and physics; big data analysis and machine ...
Meta-programming for cross-domain tensor optimizations
GPCE 2018: Proceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences

Many modern application domains crucially rely on tensor operations. The optimization of programs that operate on tensors poses difficulties that are not adequately addressed by existing languages and tools. Frameworks such as TensorFlow offer good ...

Comments

Information & Contributors

Information

Published In

cover image ACM SIGPLAN Notices

ACM SIGPLAN Notices Volume 52, Issue 12

GPCE '17

December 2017

258 pages

ISSN:0362-1340

EISSN:1558-1160

DOI:10.1145/3170492

Editor:
Matthew Fluet
Rodchester Institude of Technology

Issue’s Table of Contents

GPCE 2017: Proceedings of the 16th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences
October 2017
258 pages
ISBN:9781450355247
DOI:10.1145/3136040
General Chair:
Matthew Flatt
University of Utah, USA
,
Program Chair:
Sebastian Erdweg
TU Delft, Netherlands

Copyright © 2017 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 23 October 2017

Published in SIGPLAN Volume 52, Issue 12

Check for updates

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

8
Total Citations
View Citations
213
Total Downloads

Downloads (Last 12 months)12
Downloads (Last 6 weeks)0

Reflects downloads up to 09 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Susungi ATadonki C(2021)Intermediate Representations for Explicitly Parallel ProgramsACM Computing Surveys10.1145/345229954:5(1-24)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1145/3452299
Susungi ATadonki C(2021)Intermediate Representations for Explicitly Parallel ProgramsACM Computing Surveys10.1145/345229954:5(1-24)Online publication date: 25-May-2021
https://dl.acm.org/doi/10.1145/3452299
Susungi ARink NCohen ACastrillon JTadonki C(2020)Meta-programming for cross-domain tensor optimizationsACM SIGPLAN Notices10.1145/3393934.327813153:9(79-92)Online publication date: 7-Apr-2020
https://dl.acm.org/doi/10.1145/3393934.3278131
Rink NCastrillon JGibbons J(2019)TeIL: a type-safe imperative tensor intermediate languageProceedings of the 6th ACM SIGPLAN International Workshop on Libraries, Languages and Compilers for Array Programming10.1145/3315454.3329959(57-68)Online publication date: 8-Jun-2019
https://dl.acm.org/doi/10.1145/3315454.3329959
Susungi ARink NCohen ACastrillon JTadonki CVan Wyk ERompf T(2018)Meta-programming for cross-domain tensor optimizationsProceedings of the 17th ACM SIGPLAN International Conference on Generative Programming: Concepts and Experiences10.1145/3278122.3278131(79-92)Online publication date: 5-Nov-2018
https://dl.acm.org/doi/10.1145/3278122.3278131
Šinkarovs ABernecky RVießmann HScholz SScholz SShivers O(2018)A Rosetta Stone for array languagesProceedings of the 5th ACM SIGPLAN International Workshop on Libraries, Languages, and Compilers for Array Programming10.1145/3219753.3219754(1-10)Online publication date: 19-Jun-2018
https://dl.acm.org/doi/10.1145/3219753.3219754
Susungi AMarr SSartor J(2018)On the semantics of loop transformation languagesCompanion Proceedings of the 2nd International Conference on the Art, Science, and Engineering of Programming10.1145/3191697.3213796(230-231)Online publication date: 9-Apr-2018
https://dl.acm.org/doi/10.1145/3191697.3213796
Rink NHuismann ISusungi ACastrillon JStiller JFröhlich JTadonki C(2018)CFDlangProceedings of the Real World Domain Specific Languages Workshop 201810.1145/3183895.3183900(1-10)Online publication date: 24-Feb-2018
https://dl.acm.org/doi/10.1145/3183895.3183900

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Issue’s Table of Contents