article

High-level Language Support for User-defined Reductions

Authors:

Steven J. Deitz,

Bradford L. Chamberlain,

Lawrence SnyderAuthors Info & Claims

The Journal of Supercomputing, Volume 23, Issue 1

Pages 23 - 37

https://doi.org/10.1023/A:1015781018449

Published: 01 August 2002 Publication History

Abstract

The optimized handling of reductions on parallel supercomputers or clusters of workstations is critical to high performance because reductions are common in scientific codes and a potential source of bottlenecks. Yet in many high-level languages, a mechanism for writing efficient reductions remains surprisingly absent. Further, when such mechanisms do exist, they often do not provide the flexibility a programmer needs to achieve a desirable level of performance. In this paper, we present a new language construct for arbitrary reductions that lets a programmer achieve a level of performance equal to that achievable with the highly flexible, but low-level combination of Fortran and MPI. We have implemented this construct in the ZPL language and evaluate it in the context of the initialization of the NAS MG benchmark. We show a 45 times speedup over the same code written in ZPL without this construct. In addition, performance on a large number of processors surpasses that achieved in the NAS implementation showing that our mechanism provides programmers with the needed flexibility.

References

[1]

1. D. Bailey, E. Barszcz, J. Barton, D. Browning, R. Carter, L. Dagum, R. Fatoohi, S. Fineberg, P. Frederickson, T. Lasinski, R. Schreiber, H. Simon, V. Venkatakrishnan, and S. Weeratunga. NAS parallel benchmarks. Technical report, NASA Ames Research Center (RNR-94-007), March 1994.

[2]

2. D. Bailey, T. Harris, W. Saphir, R. van der Wijngaart, A. Woo, and M. Yarrow. NAS parallel benchmarks 2.0. Technical report. NASA Ames Research Center (NAS-95-020), December 1995.

[3]

3. G. E. Blelloch. NESL: A nested data-parallel language (Version 3.1). Technical report. Carnegie Mellon (CMU-CS-95-170), September 1995.

Digital Library

[4]

4. W. Blume, R. Eigenmann, K. Faigin, J. Grout, J. Hoeflinger, D. Padua, P. Petersen, W. Pottenger, L. Rauchwerger, P. Tu, and S. Weatherford. Polaris: Improving the effectiveness of parallelizing compilers. In Proceedings of the Workshop on Languages and Compilers for Parallel Computing, 1994.

Digital Library

[5]

5. B. L. Chamberlain, S.-E. Choi, E. C. Lewis, C. Lin, L. Snyder, and W. D. Weathersby. ZPL's WYSIWYG performance model. In Proceedings of the IEEE Workshop on High-Level Parallel Programming Models and Supportive Environments, 1998.

Digital Library

[6]

6. B. L. Chamberlain, S. J. Deitz, and L. Snyder. A comparative study of the NAS MG benchmark across parallel languages and architectures. In Proceedings of the ACM Conference on Supercomputing, 2000.

Digital Library

[7]

7. B. L. Chamberlain, E. C. Lewis, C. Lin, and L. Snyder. Regions: An abstraction for expressing array computation. In Proceedings of the ACM International Conference on Array Programming Languages, 1999.

Digital Library

[8]

8. A. L. Fisher and A. M. Ghuloum. Parallelizing complex scans and reductions. In Proceedings of the ACM Conference on Programming Language Design and Implementation, 1994.

Digital Library

[9]

9. High Performance Fortran Forum. High Performance Fortran Language Specification, Version 2.0. 1997.

[10]

10. O. Ibarra, M. C. Rinard, and P. C. Diniz. On the complexity of commutativity analysis. In Proceedings of the International Computing and Combinatorics Conference, 1996.

Digital Library

[11]

11. R. E. Ladner and M. J. Fischer. Parallel prefix computation. In Proceedings of the IEEE International Conference on Parallel Processing, 1977.

[12]

12. J. R. Larus, B. Richards, and G. Viswanathan. C**: A large-grain, object-oriented, data-parallel programming language. Technical report. University of Wisconsin-Madison (1126), November 1992.

[13]

13. J. R. Larus, B. Richards, and G. Viswanathan. Parallel programming in C**: A large-grain data-parallel programming language. In G. V. Wilson and P. Lu, eds., Parallel Programming Using C++. MIT Press, Cambridge, MA, 1996.

[14]

14. B. Lu and J. Mellor-Crummey. Compiler optimization of implicit reductions for distributed memory multiprocessors. In Proceedings of the International Parallel Processing Symposium, 1998.

Digital Library

[15]

15. M. C. Rinard and P. C. Diniz. Commutativity analysis: A new analysis framework for parallelizing compilers. In Proceedings of the ACM Conference on Programming Language Design and Implementation, 1996.

Digital Library

[16]

16. S.-B. Scholz. On defining application-specific high-level array operations by means of shape-invariant programming facilities. In Proceedings of the ACM International Conference on Array Programming Languages, 1998.

Digital Library

[17]

17. M. Snir, S. W. Otto, S. Huss-Lederman, D. W. Walker, and J. Dongarra. MPI: The Complete Reference . MIT Press, Cambridge, MA, 1996.

Digital Library

[18]

18. L. Snyder. Programming Guide to ZPL. MIT Press, Cambridge, MA, 1999.

Digital Library

[19]

19. T. Suganuma, H. Komatsu, and T. Nakatani. Detection and global optimization of reduction operations for distributed parallel machines. In Proceedings of the ACM International Conference on Supercomputing, 1996.

Digital Library

[20]

20. G. Viswanathan and J. R. Larus. User-defined reductions for efficient communication in data-parallel languages. Technical report. University of Wisconsin-Madison (1293), January 1996.

Cited By

Ginsbach PO'Boyle MReddi VSmith ATang L(2017)Discovery and exploitation of general reductions: a constraint based approachProceedings of the 2017 International Symposium on Code Generation and Optimization10.5555/3049832.3049862(269-280)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.5555/3049832.3049862
Reddy CKruse MCohen AZaks AMendelson BRauchwerger LHwu W(2016)Reduction DrawingProceedings of the 2016 International Conference on Parallel Architectures and Compilation10.1145/2967938.2967950(87-97)Online publication date: 11-Sep-2016
https://dl.acm.org/doi/10.1145/2967938.2967950
Chen JEdelman AChen JEdelman AShen WTerrel A(2014)Parallel prefix polymorphism permits parallelization, presentation & proofProceedings of the 1st First Workshop for High Performance Technical Computing in Dynamic Languages10.1109/HPTCDL.2014.9(47-56)Online publication date: 16-Nov-2014
https://dl.acm.org/doi/10.1109/HPTCDL.2014.9
Show More Cited By

Index Terms

High-level Language Support for User-defined Reductions

Recommendations

Global-view abstractions for user-defined reductions and scans
PPoPP '06: Proceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programming

Since APL, reductions and scans have been recognized as powerful programming concepts. Abstracting an accumulation loop (reduction) and an update loop (scan), the concepts have efficient parallel implementations based on the parallel prefix algorithm. ...
Parallelizing user-defined and implicit reductions globally on multiprocessors
ACSAC'06: Proceedings of the 11th Asia-Pacific conference on Advances in Computer Systems Architecture

Multiprocessors are becoming prevalent in the PC world. Major CPU vendors such as Intel and Advanced Micro Devices have migrated to multicore processors. However, this also means that computers will run an application at full speed only if that ...
A facility for the downward extension of a high-level language
Proceedings of the 1982 SIGPLAN symposium on Compiler construction

This paper presents a method whereby a high-level language can be extended to provide access to all the capabilities of the underlying hardware and operating system of a machine. In essence, it is a facility that allows a user to make special purpose ...

Comments

Information & Contributors

Information

Published In

cover image The Journal of Supercomputing

The Journal of Supercomputing Volume 23, Issue 1

August 2002

124 pages

ISSN:0920-8542

Issue’s Table of Contents

Copyright © Copyright © 2002 Kluwer Academic Publishers.

Publisher

Kluwer Academic Publishers

United States

Publication History

Published: 01 August 2002

Author Tags

Qualifiers

Article

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
0
Total Downloads

Downloads (Last 12 months)0
Downloads (Last 6 weeks)0

Reflects downloads up to 26 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ginsbach PO'Boyle MReddi VSmith ATang L(2017)Discovery and exploitation of general reductions: a constraint based approachProceedings of the 2017 International Symposium on Code Generation and Optimization10.5555/3049832.3049862(269-280)Online publication date: 4-Feb-2017
https://dl.acm.org/doi/10.5555/3049832.3049862
Reddy CKruse MCohen AZaks AMendelson BRauchwerger LHwu W(2016)Reduction DrawingProceedings of the 2016 International Conference on Parallel Architectures and Compilation10.1145/2967938.2967950(87-97)Online publication date: 11-Sep-2016
https://dl.acm.org/doi/10.1145/2967938.2967950
Chen JEdelman AChen JEdelman AShen WTerrel A(2014)Parallel prefix polymorphism permits parallelization, presentation & proofProceedings of the 1st First Workshop for High Performance Technical Computing in Dynamic Languages10.1109/HPTCDL.2014.9(47-56)Online publication date: 16-Nov-2014
https://dl.acm.org/doi/10.1109/HPTCDL.2014.9
Sathe SShrimankar DJena SKumar RTuruk ADash M(2011)Parallelization of DNA sequence alignment using OpenMPProceedings of the 2011 International Conference on Communication, Computing & Security10.1145/1947940.1947983(200-203)Online publication date: 12-Feb-2011
https://dl.acm.org/doi/10.1145/1947940.1947983
Breitbart J(2010)An approach for semiautomatic locality optimizations using OpenMPProceedings of the 10th international conference on Applied Parallel and Scientific Computing - Volume 210.1007/978-3-642-28145-7_29(291-301)Online publication date: 6-Jun-2010
https://dl.acm.org/doi/10.1007/978-3-642-28145-7_29
Duran AFerrer RKlemm Mde Supinski BAyguadé E(2010)A proposal for user-defined reductions in OpenMPProceedings of the 6th international conference on Beyond Loop Level Parallelism in OpenMP: accelerators, Tasking and more10.1007/978-3-642-13217-9_4(43-55)Online publication date: 14-Jun-2010
https://dl.acm.org/doi/10.1007/978-3-642-13217-9_4
Gan GWang XManzano JGao G(2009)Tile ReductionProceedings of the 5th International Workshop on OpenMP: Evolving OpenMP in an Age of Extreme Parallelism10.1007/978-3-642-02303-3_12(140-153)Online publication date: 22-May-2009
https://dl.acm.org/doi/10.1007/978-3-642-02303-3_12
Snyder LRyder BHailpern B(2007)The design and development of ZPLProceedings of the third ACM SIGPLAN conference on History of programming languages10.1145/1238844.1238852(8-1-8-37)Online publication date: 9-Jun-2007
https://dl.acm.org/doi/10.1145/1238844.1238852
Deitz SCallahan DChamberlain BSnyder LTorrellas JChatterjee S(2006)Global-view abstractions for user-defined reductions and scansProceedings of the eleventh ACM SIGPLAN symposium on Principles and practice of parallel programming10.1145/1122971.1122980(40-47)Online publication date: 29-Mar-2006
https://dl.acm.org/doi/10.1145/1122971.1122980
van der Mark PWolters LCats GHaddad HOmicini AWainwright RLiebrock L(2004)Using semi-lagrangian formulations with automatic code generation for environmental modelingProceedings of the 2004 ACM symposium on Applied computing10.1145/967900.967950(229-234)Online publication date: 14-Mar-2004
https://dl.acm.org/doi/10.1145/967900.967950
Show More Cited By

View Options

View options

Media

Figures

Other

Tables

View Issue’s Table of Contents