Compiling Recurrences over Dense and Sparse Arrays
Article No.: 103, Pages 250 - 275
Abstract
We present a framework for compiling recurrence equations into native code. In our framework, users specify a system of recurrences, the types of data structures that store inputs and outputs, and scheduling commands for optimization. Our compiler then lowers these specifications into native code that respects the dependencies in the recurrence equations. Our compiler can generate code over both sparse and dense data structures, and determines if the recurrence system is solvable with the provided scheduling primitives. We evaluate the performance and correctness of the generated code on several recurrences, from domains as diverse as dense and sparse matrix solvers, dynamic programming, graph problems, and sparse tensor algebra. We demonstrate that the generated code has competitive performance to hand-optimized implementations in libraries. However, these handwritten libraries target specific recurrences, specific data structures, and specific optimizations. Our system, on the other hand, automatically generates implementations from recurrences, data formats, and schedules, giving our system more generality than library approaches.
References
[1]
Richard E Bellman. 2010. Dynamic programming. Princeton university press.
[2]
Aart J.C. Bik, Bixia Zheng, Fredrik Kjolstad, Nicolas Vasilache, Penporn Koanantakool, and Tatiana Shpeisman. 2022. Compiler Support for Sparse Tensor Computations in MLIR. ACM Transactions on Architecture and Code Optimization.
[3]
Boost. 2002. The Boost Graph Library: User Guide and Reference Manual. Addison-Wesley Longman Publishing Co., Inc., USA. isbn:0201729148
[4]
Gaurav Chaurasia, Jonathan Ragan-Kelley, Sylvain Paris, George Drettakis, and Fredo Durand. 2015. Compiling high performance recursive filters. In Proceedings of the 7th conference on high-performance graphics. 85–94.
[5]
Kazem Cheshmi, Shoaib Kamil, Michelle Mills Strout, and Maryam Mehri Dehnavi. 2017. Sympiler: transforming sparse matrix codes by decoupling symbolic analysis. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. 1–13.
[6]
Stephen Chou, Fredrik Kjolstad, and Saman Amarasinghe. 2018. Format Abstraction for Sparse Tensor Algebra Compilers. Proc. ACM Program. Lang., 2, OOPSLA (2018), Article 123, oct, 30 pages. https://doi.org/10.1145/3276493
[7]
Jeffrey A. Daily. 2016. Parasail: SIMD C library for global, semi-global, and local pairwise sequence alignments. BMC Bioinformatics, 17 (2016), 2, https://doi.org/10.1186/s12859-016-0930-z
[8]
T. A. Davis. 2006. CXSparse: a Concise eXtended Sparse Matrix Package. https://github.com/DrTimothyAldenDavis/SuiteSparse/tree/dev/CXSparse
[9]
T. A. Davis. 2006. Direct Methods for Sparse Linear Systems. SIAM, Philadelphia, PA.
[10]
Jason Eisner, Eric Goldlust, and Noah A. Smith. 2004. Dyna: a declarative language for implementing dynamic programs. In Annual Meeting of the Association for Computational Linguistics.
[11]
Paul Feautrier. 1991. Dataflow Analysis of Array and Scalar References. International Journal of Parallel Programming, 20, 1 (1991), 23–53.
[12]
Gautam and S. Rajopadhye. 2006. Simplifying Reductions. SIGPLAN Not., 41, 1 (2006), jan, 30–41. issn:0362-1340
[13]
John R Gilbert. 1994. Predicting structure in sparse matrix computations. SIAM J. Matrix Anal. Appl., 15, 1 (1994), 62–79.
[14]
John A. Gunnels, Fred G. Gustavson, Greg M. Henry, and Robert A. van de Geijn. 2001. FLAME: Formal Linear Algebra Methods Environment. ACM Trans. Math. Softw., 27, 4 (2001), dec, 422–455. issn:0098-3500 https://doi.org/10.1145/504210.504213
[15]
Rawn Henry, Olivia Hsu, Rohan Yadav, Stephen Chou, Kunle Olukotun, Saman Amarasinghe, and Fredrik Kjolstad. 2021. Compilation of sparse array programming models. Proceedings of the ACM on Programming Languages, 5, OOPSLA (2021), 1–29.
[16]
Liang Huang. 2008. Advanced dynamic programming in semiring and hypergraph frameworks. Coling 2008: Advanced Dynamic Programming in Computational Linguistics: Theory, Algorithms and Applications-Tutorial notes, 1–18.
[17]
Richard M Karp, Raymond E Miller, and Shmuel Winograd. 1967. The Organization of Computations for Uniform Recurrence Equations. J. ACM, 14, 3 (1967), 563–590.
[18]
Jeremy Kepner, Peter Aaltonen, David Bader, Aydin Buluç, Franz Franchetti, John Gilbert, Dylan Hutchison, Manoj Kumar, Andrew Lumsdaine, and Henning Meyerhenke. 2016. Mathematical foundations of the GraphBLAS. In 2016 IEEE High Performance Extreme Computing Conference (HPEC). 1–9.
[19]
Fredrik Kjolstad, Peter Ahrens, Shoaib Kamil, and Saman Amarasinghe. 2019. Tensor Algebra Compilation with Workspaces. In Proceedings of the 2019 IEEE/ACM International Symposium on Code Generation and Optimization (CGO 2019). IEEE Press, 180–192. isbn:9781728114361
[20]
Fredrik Kjolstad, Shoaib Kamil, Stephen Chou, David Lugato, and Saman P. Amarasinghe. 2017. The tensor algebra compiler. Proc. ACM Program. Lang., 1, OOPSLA (2017), 77:1–77:29.
[21]
Fredrik Berg Kjølstad. 2020. Sparse tensor algebra compilation. Ph. D. Dissertation. Massachusetts Institute of Technology.
[22]
Robert Kowalski and Keith L Clark. 2003. Logic programming. In Encyclopedia of Computer Science. 1017–1031.
[23]
Leslie Lamport. 1974. The Parallel Execution of DO Loops. Commun. ACM, 17, 2 (1974), 83–93.
[24]
R Sherman Lehman. 1960. DYNAMIC PROGRAMMING AND GAUSSIAN ELIMINATION. RAND CORP SANTA MONICA CALIF.
[25]
Joseph W Liu. 1986. A compact row storage scheme for Cholesky factors using elimination trees. ACM Transactions on Mathematical Software (TOMS), 12, 2 (1986), 127–148.
[26]
Joseph W. H. Liu. 1992. The Multifrontal Method for Sparse Matrix Solution: Theory and Practice. SIAM Rev., 34 (1992), 82–109.
[27]
Vijay Menon and Keshav Pingali. 2004. Look left, look right, look left again: An application of fractal symbolic analysis to linear algebra code restructuring. International Journal of Parallel Programming, 32 (2004), 501–523.
[28]
Esmond Ng and Barry W. Peyton. 1993. A Supernodal Cholesky Factorization Algorithm for Shared-Memory Multiprocessors. SIAM Journal on Scientific Computing, 14, 4 (1993), 761–769. https://doi.org/10.1137/0914048 arxiv:https://doi.org/10.1137/0914048.
[29]
James M. Ortega. 1988. The ijk forms of factorization methods I. Vector computers. Parallel Comput., 7 (1988), 135–147.
[30]
James M. Ortega. 1988. The ijk forms of factorization methods II. Vector computers. Parallel Comput., 7 (1988), 149–162.
[31]
Louis-Noel Pouchet. 2016. PolyBench. http://web.cs.ucla.edu/~pouchet/software/polybench/
[32]
Jonathan Ragan-Kelley, Connelly Barnes, Andrew Adams, Sylvain Paris, Frédo Durand, and Saman P. Amarasinghe. 2013. Halide: a language and compiler for optimizing parallelism, locality, and recomputation in image processing pipelines. In ACM SIGPLAN Conference on Programming Language Design and Implementation, PLDI ’13, Seattle, WA, USA, June 16-19, 2013, Hans-Juergen Boehm and Cormac Flanagan (Eds.). ACM, 519–530.
[33]
MMG Ricci and Tullio Levi-Civita. 1900. Méthodes de calcul différentiel absolu et leurs applications. Math. Ann., 54, 1-2 (1900), 125–201.
[34]
Georg Sauthoff, Mathias Möhl, Stefan Janssen, and Robert Giegerich. 2013. Bellman’s GAP—a language and compiler for dynamic programming in sequence analysis. Bioinformatics, 29, 5 (2013), 01, 551–560. issn:1367-4803 arxiv:https://academic.oup.com/bioinformatics/article-pdf/29/5/551/16919063/btt022.pdf.
[35]
Ryan Senanayake, Changwan Hong, Ziheng Wang, Amalee Wilson, Stephen Chou, Shoaib Kamil, Saman Amarasinghe, and Fredrik Kjolstad. 2020. A Sparse Iteration Space Transformation Framework for Sparse Tensor Algebra. Proc. ACM Program. Lang., 4, OOPSLA (2020), Article 158, nov, 30 pages. https://doi.org/10.1145/3428226
[36]
Michelle Mills Strout, Mary Hall, and Catherine Olschanowsky. 2018. The sparse polyhedral framework: Composing compiler-generated inspector-executor code. Proc. IEEE, 106, 11 (2018), 1921–1934.
[37]
Shiv Sundram, Muhammad Usman Tariq, and Fredrik Kjolstad. 2024. Artifact for OOPSLA 2024 Paper: Compiling Recurrences over Dense and Sparse Arrays (version 1). https://doi.org/10.5281/zenodo.10774458
[38]
Ruiqin Tian, Luanzheng Guo, Jiajia Li, Bin Ren, and Gokcen Kestor. 2021. A High Performance Sparse Tensor Algebra Compiler in MLIR. 12.
[39]
A. Viterbi. 1967. Error bounds for convolutional codes and an asymptotically optimum decoding algorithm. IEEE Transactions on Information Theory, 13, 2 (1967), 260–269. https://doi.org/10.1109/TIT.1967.1054010
[40]
Cambridge Yang, Eric Atkinson, and Michael Carbin. 2021. Simplifying Dependent Reductions in the Polyhedral Model. Proc. ACM Program. Lang., 5, POPL (2021), Article 20, jan, 33 pages.
[41]
Zihao Ye, Ruihang Lai, Junru Shao, Tianqi Chen, and Luis Ceze. 2023. SparseTIR: Composable abstractions for sparse compilation in deep learning. In Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 3. 660–678.
Index Terms
- Compiling Recurrences over Dense and Sparse Arrays
Recommendations
Compiler Support for Sparse Tensor Convolutions
This paper extends prior work on sparse tensor algebra compilers to generate asymptotically efficient code for tensor expressions with affine subscript expressions. Our technique enables compiler support for a wide range of sparse computations, including ...
Compilation of Modular and General Sparse Workspaces
Recent years have seen considerable work on compiling sparse tensor algebra expressions. This paper addresses a shortcoming in that work, namely how to generate efficient code (in time and space) that scatters values into a sparse result tensor. We ...
Compilation of dynamic sparse tensor algebra
Many applications, from social network graph analytics to control flow analysis, compute on sparse data that evolves over the course of program execution. Such data can be represented as dynamic sparse tensors and efficiently stored in formats (data ...
Comments
Information & Contributors
Information
Published In
April 2024
1492 pages
Copyright © 2024 Owner/Author.
This work is licensed under a Creative Commons Attribution International 4.0 License.
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 29 April 2024
Published in PACMPL Volume 8, Issue OOPSLA1
Check for updates
Badges
Author Tags
Qualifiers
- Research-article
Funding Sources
- SRC
- NSF (National Science Foundation)
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 391Total Downloads
- Downloads (Last 12 months)391
- Downloads (Last 6 weeks)59
Reflects downloads up to 10 Feb 2025
Other Metrics
Citations
View Options
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign in