research-article

Open access

Simplifying dependent reductions in the polyhedral model

Authors:

Cambridge Yang,

Michael CarbinAuthors Info & Claims

Proceedings of the ACM on Programming Languages, Volume 5, Issue POPL

Article No.: 20, Pages 1 - 33

https://doi.org/10.1145/3434301

Published: 04 January 2021 Publication History

Abstract

A Reduction – an accumulation over a set of values, using an associative and commutative operator – is a common computation in many numerical computations, including scientific computations, machine learning, computer vision, and financial analytics. Contemporary polyhedral-based compilation techniques make it possible to optimize reductions, such as prefix sums, in which each component of the reduction’s output potentially shares computation with another component in the reduction. Therefore an optimizing compiler can identify the computation shared between multiple components and generate code that computes the shared computation only once.

These techniques, however, do not support reductions that – when phrased in the language of the polyhedral model – span multiple dependent statements. In such cases, existing approaches can generate incorrect code that violates the data dependences of the original, unoptimized program.

In this work, we identify and formalize the optimization of dependent reductions as an integer bilinear program. We present a heuristic optimization algorithm that uses an affine sequential schedule of the program to determine how to simplfy reductions yet still preserve the program’s dependences.

We demonstrate that the algorithm provides optimal complexity for a set of benchmark programs from the literature on probabilistic inference algorithms, whose performance critically relies on simplifying these reductions. The complexities for 10 of the 11 programs improve siginifcantly by factors at least of the sizes of the input data, which are in the range of 10⁴ to 10⁶ for typical real application inputs. We also confirm the significance of the improvement by showing speedups in wall-clock time that range from 1.1x to over 10⁶x.

Supplementary Material

Auxiliary Archive (popl21main-p145-p-archive.zip)

Appendices to the paper.

Download
423.48 KB

References

[1]

Christophe Alias, Alain Darte, Paul Feautrier, and Laure Gonnord. 2010. Multi-dimensional Rankings, Program Termination, and Complexity Bounds of Flowchart Programs. In Static Analysis Symposium. https://hal.inria.fr/inria-00523298

[2]

Eric Atkinson, Cambridge Yang, and Michael Carbin. 2018. Verifying Handcoded Probabilistic Inference Procedures. In arXiv e-prints.

[3]

Stefan Behnel, Robert Bradshaw, Craig Citro, Lisandro Dalcin, Dag Sverre Seljebotn, and Kurt Smith. 2011. Cython: The Best of Both Worlds. Computing in Science and Engg. 13, 2 (March 2011 ), 31-39.

Digital Library

[4]

Mohamed-Walid Benabderrahmane, Louis-Noël Pouchet, Albert Cohen, and Cédric Bastoul. 2010. The Polyhedral Model is More Widely Applicable Than You Think. In European Conference on Theory and Practice of Software, International Conference on Compiler Construction.

Digital Library

[5]

Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rohit Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2019. Pyro: Deep Universal Probabilistic Programming. Journal of Machine Learning Research 20, 28 ( 2019 ).

[6]

Christopher M. Bishop. 2006. Pattern Recognition and Machine Learning ( Information Science and Statistics). Springer-Verlag, Berlin, Heidelberg.

[7]

David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003. Latent Dirichlet Allocation. Journal of Machine Learning Research 3 (Jan. 2003 ), 993-1022.

Digital Library

[8]

Uday Bondhugula, Albert Hartono, J. Ramanujam, and P. Sadayappan. 2008. A Practical Automatic Polyhedral Parallelizer and Locality Optimizer. In Conference on Programming Language Design and Implementation.

[9]

Jean-François Collard, Denis Barthou, and Paul Feautrier. 1995. Fuzzy Array Dataflow Analysis. In Symposium on Principles and Practice of Parallel Programming.

[10]

D Collett. 1993. Modelling Survival Data in Medical Research. Chapman & Hall, New York.

[11]

D. R. Cox. 1972. Regression Models and Life-Tables. Journal of the Royal Statistical Society: Series B (Methodological) 34, 2 ( 1972 ), 187-202.

[12]

Marco F Cusumano-Towner, Feras A Saad, Alexander K Lew, and Vikash K Mansinghka. 2019. Gen: a General-purpose Probabilistic Programming System with Programmable Inference. In Conference on Programming Language Design and Implementation.

Digital Library

[13]

Greg Morisett Daniel Huang, Jean-Baptiste Tristan. 2017. Compiling Markov Chain Monte Carlo Algorithms for Probabilistic Modeling. In Conference on Programming Language Design and Implementation.

[14]

Johannes Doerfert, Kevin Streit, Sebastian Hack, and Zino Benaissa. 2015. Polly's Polyhedral Scheduling in the Presence of Reductions. In International Workshop on Polyhedral Compilation Techniques.

[15]

E. Ehrhardt. 1967. Sur un problème de Géométrie Diophantienne Linéaire. II. Journal für die Reine und Angewandte Mathematik 1967 ( 1967 ), 25-49. Issue 227.

[16]

P. Feautrier. 1988. Array Expansion. In International Conference on Supercomputing.

[17]

Paul Feautrier. 1992a. Some eficient solutions to the afine scheduling problem. I. One-dimensional time. International Journal of Parallel Programming 21, 5 (Oct. 1992 ), 313-347.

[18]

Paul Feautrier. 1992b. Some eficient solutions to the afine scheduling problem. Part II. Multidimensional time. International Journal of Parallel Programming 21, 6 (Dec. 1992 ), 389-420.

[19]

Robert M. Fung and Kuo-Chu Chang. 1989. Weighing and Integrating Evidence for Stochastic Simulation on Bayesian Networks. In Conference on Uncertainty in Artificial Intelligence.

[20]

Gautam and S. Rajopadhye. 2006. Simplifying Reductions. In Symposium on Principles of Programming Languages.

[21]

Andrew Gelman, Daniel Lee, and Jiqiang Guo. 2015. Stan: A probabilistic programming language for Bayesian inference and optimization. Journal of Educational and Behavioral Statistics 40, 5 ( 2015 ), 530-543.

[22]

Stuart Geman and Donald Geman. 1984. Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of images. Transactions on Pattern Analysis and Machine Intelligence 6 ( Nov. 1984 ), 721-741. Issue 6.

Digital Library

[23]

Philip Ginsbach and Michael F. P. O'Boyle. 2017. Discovery and Exploitation of General Reductions: A Constraint Based Approach. In International Symposium on Code Generation and Optimization.

[24]

Noah D. Goodman, Vikash K. Mansinghka, Daniel M. Roy, Keith Bonawitz, and Joshua B. Tenenbaum. 2008. Church: A language for generative models. In Conference on Uncertainty in Artificial Intelligence.

[25]

Noah D Goodman and Andreas Stuhlmüller. 2014. The Design and Implementation of Probabilistic Programming Languages. http://dippl.org. Accessed: 2020-10-30.

[26]

Scott Grauer-Gray and John Cavazos. 2011. Optimizing and Auto-tuning Belief Propagation on the GPU. In Workshop on Languages and Compilers for Parallel Computing.

[27]

T. Grifiths and M. Steyvers. 2004. Finding Scientific Topics. Proceedings of the National Academy of Sciences 101, suppl. 1 (April 2004 ), 5228-5235.

[28]

Gautam Gupta, Kim Daegon, and Sanjay Rajopadhye. 2007. Scheduling in the Z-Polyhedral Model. International Parallel and Distributed Processing Symposium.

[29]

Gautam Gupta, Sanjay Rajopadhye, and Patrice Quinton. 2002. Scheduling Reductions on Realistic Machines. In Symposium on Parallel Algorithms and Architectures.

[30]

W. K. Hastings. 1970. Monte Carlo Sampling Methods Using Markov Chains and Their Applications. Biometrika 57, 1 (April 1970 ), 97-109.

[31]

Ian Holmes, Keith Harris, and Christopher Quince. 2012. Dirichlet Multinomial Mixtures: Generative Models for Microbial Metagenomics. PLOS ONE ( 2012 ).

[32]

Guillaume Iooss, Christophe Alias, and Sanjay Rajopadhye. 2014. On Program Equivalence with Reductions. In International Static Analysis Symposium.

[33]

Jian Sun, Nan-Ning Zheng, and Heung-Yeung Shum. 2003. Stereo matching using belief propagation. Transactions on Pattern Analysis and Machine Intelligence 25, 7 ( July 2003 ), 787-800.

[34]

Ryoichi Kikuchi. 1951. A Theory of Cooperative Phenomena. Physical Review 81 (March 1951 ), 988-1003. Issue 6.

[35]

Oleg Kiselyov. 2016. Probabilistic Programming Language and its Incremental Evaluation. In Asian Symposium on Programming Languages and Systems.

[36]

Jun S. Liu. 1994. The Collapsed Gibbs Sampler in Bayesian Computations with Applications to a Gene Regulation Problem. J. Amer. Statist. Assoc. 89, 427 (Sept. 1994 ), 958-966.

[37]

Yanhong A. Liu, Scott D. Stoller, Ning Li, and Tom Rothamel. 2005. Optimizing Aggregate Array Computations in Loops. ACM Transactions on Programming Languages and Systems 27, 1 (Jan. 2005 ), 91-125.

Digital Library

[38]

Vikash Mansingkha, Ulrich Schaechtle, Shivam Handa, Alexey Radul, Yutian Chen, and Martin Rinard. 2018. Probabilistic Programming with Programmable Inference. In Conference on Programming Language Design and Implementation.

[39]

N. Metropolis, A. W. Rosenbluth, M. N. Rosenbluth, A. H. Teller, and E. Teller. 1953. Equation of State Calculations by Fast Computing Machines. Journal of Chemical Physics 21, 6 ( 1953 ), 1087-1092.

[40]

Kevin P. Murphy. 2012. Machine Learning: A Probabilistic Perspective. MIT Press, Cambridge, Massachusets.

Digital Library

[41]

Praveen Narayanan, Jacques Carette, Wren Romano, Chung-chieh Shan, and Robert Zinkov. 2016. Probabilistic Inference by Program Transformation in Hakaru (System Description). In International Symposium on Functional and Logic Programming.

[42]

George L. Nemhauser and Laurence A. Wolsey. 1988. Integer and Combinatorial Optimization.

Digital Library

[43]

David Newman. 2008. Bag of Words Dataset. In UCI Machine Learning Respository.

[44]

Aditya V. Nori, Sherjil Ozair, Sriram K. Rajamani, and Deepak Vijaykeerthy. 2015. Eficient Synthesis of Probabilistic Programs. In Conference on Programming Language Design and Implementation.

[45]

David Padua (Ed.). 2011. Omega Calculator. Springer US, Boston, MA, 1355-1355. https://doi.org/10.1007/978-0-387-09766-4_2303

[46]

Martyn Plummer. 2015. JAGS Version 4.0.0 user manual. Addison-Wesley, Reading, Massachusetts.

[47]

Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and John Cavazos. 2008. Iterative optimization in the polyhedral model: Part II, multidimensional time.

[48]

Louis-Noël Pouchet, Cédric Bastoul, Albert Cohen, and Nicolas Vasilache. 2007. Iterative optimization in the polyhedral model: Part I, one-dimensional time. In International Symposium on Code Generation and Optimization.

[49]

Louis-Noël Pouchet, Uday Bondhugula, Cédric Bastoul, Albert Cohen, J. Ramanujam, P. Sadayappan, and Nicolas Vasilache. 2011. Loop Transformations: Convexity, Pruning and Optimization. In Symposium on Principles of Programming Languages.

[50]

L. Rauchwerger and D. A. Padua. 1999. The LRPD test: speculative run-time parallelization of loops with privatization and reduction parallelization. Transactions on Parallel and Distributed Systems 10, 2 (Feb. 1999 ), 160-180.

Digital Library

[51]

C. Reddy, M. Kruse, and A. Cohen. 2016. Reduction drawing: Language constructs and polyhedral compilation for reductions on GPUs.

[52]

Xavier Redon and Paul Feautrier. 1994. Scheduling Reductions. In International Conference on Supercomputing.

[53]

Philip Resnik and Eric Hardisty. 2010. Gibbs Sampling for the Uninitiated. Technical Report.

[54]

Daniel Ritchie, Andreas Stuhlmüller, and Noah Goodman. 2016. C3: Lightweight Incrementalized MCMC for Probabilistic Programs using Continuations and Callsite Caching. In International Conference on Artificial Intelligence and Statistics.

[55]

Thomas Rubiano. 2017. Implicit Computational Complexity and Compilers. Ph.D. Dissertation.

[56]

Yannick Saouter and Patrice Quinton. 1993. Computability of Recurrence Equations. Theoretical Computer Science 116, 2 (Aug. 1993 ), 317-337.

Digital Library

[57]

Alexander Schrijver. 1986. Theory of Linear and Integer Programming. John Wiley & Sons, Inc., New York, NY, USA.

Digital Library

[58]

G. A. Susto, A. Schirru, S. Pampuri, S. McLoone, and A. Beghi. 2015. Machine Learning for Predictive Maintenance: A Multiple Classifier Approach. Transactions on Industrial Informatics 11, 3 ( June 2015 ), 812-820.

[59]

Patricia M Therneau, Terry M. ; Grambsch. 2013. Modeling Survival Data: Extending the Cox Model. Springer, New York.

[60]

Dustin Tran, Matthew D Hofman, Rif A Saurous, Eugene Brevdo, Kevin Murphy, and David M Blei. 2017. Deep probabilistic programming. In International Conference on Learning Representations.

[61]

Peter Turnbaugh, Micah Hamady, Tanya Yatsunenko, Brandi Cantarel, Alexis Duncan, Ruth Ley, Mitchell Sogin, Joe Jones, Bruce A Roe, Jason Afourtit, Michael Egholm, Bernard Henrissat, Andrew C Heath, Rob Knight, and Jefrey I Gordon. 2008. A core gut microbiome in obese and lean twins. Nature 457 (12 2008 ), 480-4.

[62]

Sven Verdoolaege. 2010. isl: An Integer Set Library for the Polyhedral Model. In International Congress on Mathematical Software.

[63]

Sven Verdoolaege. 2016. Presburger Formulas and Polyhedral Compilation. https://doi.org/10.13140 /RG.2.1.1174.6323

[64]

Sven Verdoolaege, Hristo Nikolov, and Todor Stefanov. 2013. On Demand Parametric Array Dataflow Analysis. In International Workshop on Polyhedral Compilation Techniques.

[65]

Sven Verdoolaege, Rachid Seghir, Kristof Beyls, Vincent Loechner, and Maurice Bruynooghe. 2007. Counting Integer Points in Parametric Polytopes Using Barvinok's Rational Functions. (May 2007 ).

[66]

Rajan Walia, Praveen Narayanan, Jacques Carette, Sam Tobin-Hochstadt, and Chung-chieh Shan. 2019. From High-level Inference Algorithms to Eficient Code. In International Conference on Functional Programming.

[67]

Nicola White, Fiona Reid, Adam Harris, Priscilla Harries, and Patrick Stone. 2016. A Systematic Review of Predictions of Survival in Palliative Care: How Accurate Are Clinicians and Who Are the Experts? PLOS ONE ( 08 2016 ).

[68]

Yi Wu, Lei Li, Stuart Russell, and Rastislav Bodik. 2016. In International Joint Conferences on Artificial Intelligence.

[69]

Lingfeng Yang, Patrick Hanrahan, and Noah Goodman. 2014. Generating Eficient MCMC Kernels from Probabilistic Programs. In International Conference on Artificial Intelligence and Statistics.

[70]

Tomofumi Yuki, Gautam Gupta, DaeGon Kim, Tanveer Pathan, and Sanjay Rajopadhye. 2013. AlphaZ: A System for Design Space Exploration in the Polyhedral Model. In Workshop on Languages and Compilers for Parallel Computing.

[71]

Jieyuan Zhang and Jingling Xue. 2019. Incremental Precision-Preserving Symbolic Inference for Probabilistic Programs. In Conference on Programming Language Design and Implementation.

Cited By

Rasch A(2024)(De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional HomomorphismsACM Transactions on Programming Languages and Systems10.1145/366564346:3(1-74)Online publication date: 10-Oct-2024
https://dl.acm.org/doi/10.1145/3665643
Sundram STariq MKjolstad F(2024)Compiling Recurrences over Dense and Sparse ArraysProceedings of the ACM on Programming Languages10.1145/36498208:OOPSLA1(250-275)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649820
Rasch ASchulze RShabalin DElster AGorlatch SHall MVerbrugge CLhoták OShen X(2023)(De/Re)-Compositions Expressed Systematically via MDH-Based SchedulesProceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction10.1145/3578360.3580269(61-72)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3578360.3580269
Show More Cited By

Index Terms

Simplifying dependent reductions in the polyhedral model
1. Software and its engineering
  1. Software notations and tools
    1. Compilers
2. Theory of computation
  1. Models of computation
    1. Probabilistic computation
  2. Semantics and reasoning
    1. Program reasoning
      1. Program analysis
    2. Program semantics

Recommendations

Simplifying reductions
Proceedings of the 2006 POPL Conference

We present optimization techniques for high level equational programs that are generalizations of affine control loops (ACLs). Significant parts of the SpecFP and PerfectClub benchmarks are ACLs. They often contain reductions: associative and ...
Reductions for safety proofs

Program reductions are used widely to simplify reasoning about the correctness of concurrent and distributed programs. In this paper, we propose a general approach to proof simplification of concurrent programs based on exploring generic classes of ...
Reductions between disjoint NP-Pairs

Disjoint NP-pairs are pairs (A, B) of nonempty, disjoint sets in NP. We prove that all of the following assertions are equivalent: There is a many-one complete disjoint NP-pair; there is a strongly many-one complete disjoint NP-pair; there is a Turing ...

Comments

Information & Contributors

Information

Published In

cover image Proceedings of the ACM on Programming Languages

Proceedings of the ACM on Programming Languages Volume 5, Issue POPL

January 2021

1789 pages

EISSN:2475-1421

DOI:10.1145/3445980

Issue’s Table of Contents

Copyright © 2021 Owner/Author.

This work is licensed under a Creative Commons Attribution-NoDerivatives International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 04 January 2021

Published in PACMPL Volume 5, Issue POPL

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Office of Naval Research

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

13
Total Citations
View Citations
891
Total Downloads

Downloads (Last 12 months)206
Downloads (Last 6 weeks)21

Reflects downloads up to 10 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Rasch A(2024)(De/Re)-Composition of Data-Parallel Computations via Multi-Dimensional HomomorphismsACM Transactions on Programming Languages and Systems10.1145/366564346:3(1-74)Online publication date: 10-Oct-2024
https://dl.acm.org/doi/10.1145/3665643
Sundram STariq MKjolstad F(2024)Compiling Recurrences over Dense and Sparse ArraysProceedings of the ACM on Programming Languages10.1145/36498208:OOPSLA1(250-275)Online publication date: 29-Apr-2024
https://dl.acm.org/doi/10.1145/3649820
Rasch ASchulze RShabalin DElster AGorlatch SHall MVerbrugge CLhoták OShen X(2023)(De/Re)-Compositions Expressed Systematically via MDH-Based SchedulesProceedings of the 32nd ACM SIGPLAN International Conference on Compiler Construction10.1145/3578360.3580269(61-72)Online publication date: 17-Feb-2023
https://dl.acm.org/doi/10.1145/3578360.3580269
Brauckmann APolgreen EGrosser TO'Boyle M(2023)mlirSynth: Automatic, Retargetable Program Raising in Multi-Level IR Using Program Synthesis2023 32nd International Conference on Parallel Architectures and Compilation Techniques (PACT)10.1109/PACT58117.2023.00012(39-50)Online publication date: 21-Oct-2023
https://doi.org/10.1109/PACT58117.2023.00012
Huang YGuo SDi SLi GCappello FWolf FShende SCulhane CAlam SJagode H(2022)Mitigating silent data corruptions in HPC applications across multiple program inputsProceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis10.5555/3571885.3571907(1-14)Online publication date: 13-Nov-2022
https://dl.acm.org/doi/10.5555/3571885.3571907
Sun BYang LHajiesmaili MWierman ALui JTowsley DTsang D(2022)The Online Knapsack Problem with DeparturesProceedings of the ACM on Measurement and Analysis of Computing Systems10.1145/35706186:3(1-32)Online publication date: 8-Dec-2022
https://dl.acm.org/doi/10.1145/3570618
Zhao JBastoul CYi YHu JNie WZhang RGeng ZLi CTachon TGan ZKloeckner AMoreira J(2022)Parallelizing Neural Network Models Effectively on GPU by Implementing Reductions AtomicallyProceedings of the International Conference on Parallel Architectures and Compilation Techniques10.1145/3559009.3569656(451-466)Online publication date: 8-Oct-2022
https://dl.acm.org/doi/10.1145/3559009.3569656
Yang LZeynali AHajiesmaili MSitaraman RTowsley D(2022)Competitive Algorithms for Online Multidimensional Knapsack ProblemsACM SIGMETRICS Performance Evaluation Review10.1145/3547353.352262750:1(87-88)Online publication date: 20-Jun-2022
https://dl.acm.org/doi/10.1145/3547353.3522627
Yang LNie BJog ASmirni E(2022)SUGAR: Speeding Up GPGPU Application Resilience Estimation with Input SizingACM SIGMETRICS Performance Evaluation Review10.1145/3543516.345391749:1(45-46)Online publication date: 7-Jun-2022
https://dl.acm.org/doi/10.1145/3543516.3453917
Yang LZeynali AHajiesmaili MSitaraman RTowsley DManjunath DNair JCarlsson NCohen ERobert P(2022)Competitive Algorithms for Online Multidimensional Knapsack ProblemsAbstract Proceedings of the 2022 ACM SIGMETRICS/IFIP PERFORMANCE Joint International Conference on Measurement and Modeling of Computer Systems10.1145/3489048.3522627(87-88)Online publication date: 6-Jun-2022
https://dl.acm.org/doi/10.1145/3489048.3522627
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Article

Media

Figures

Other

Tables

View Issue’s Table of Contents