research-article

Open access

Library Development with MPI: Attributes, Request Objects, Group Communicator Creation, Local Reductions, and Datatypes

Authors:

Jesper Larsson Träff,

Ioannis VardasAuthors Info & Claims

EuroMPI '23: Proceedings of the 30th European MPI Users' Group Meeting

Article No.: 5, Pages 1 - 10

https://doi.org/10.1145/3615318.3615323

Published: 21 September 2023 Publication History

All formats PDF

Abstract

A major design objective of MPI is to enable support for the construction of safe parallel libraries that can be used and mixed freely in complex applications. In this respect, MPI has been extremely successful; but may nevertheless lack elementary supporting functionality for some situations, and may have made design choices that are difficult to accommodate in certain libraries. We discuss several cases of library construction requiring different kinds of supporting MPI functionality, and propose concrete improvements for library implementations and future MPI versions to alleviate the problems that were encountered. Specifically, we pinpoint (performance) issues with MPI object attributes, caching and lookup, request objects, partly collective and non-blocking communicator creation, process local reductions, type correct process local copying, and user-defined datatypes.

References

[1]

Enes Bajrović and Jesper Larsson Träff. 2011. Using MPI derived datatypes in numerical libraries. In Recent Advances in Message Passing Interface. 18th European MPI Users’ Group Meeting(Lecture Notes in Computer Science, Vol. 6960). Springer, 29–38.

[2]

Pavan Balaji, Darius Buntinas, Satish Balay, Barry F. Smith, Rajeev Thakur, and William Gropp. 2007. Nonuniformly Communicating Noncontiguous Data: A Case Study with PETSc and MPI. In 21th International Parallel and Distributed Processing Symposium (IPDPS). 1–10.

[3]

Pavan Balaji, Anthony Chan, William Gropp, Rajeev Thakur, and Ewing L. Lusk. 2010. The Importance of Non-Data-Communication Overheads in MPI. International Journal on High Performance Computing Applications 24, 1 (2010), 5–15.

Digital Library

[4]

Alexandra Carpen-Amarie, Sascha Hunold, and Jesper Larsson Träff. 2017. On Expected and Observed Communication Performance with MPI Derived Datatypes. Parallel Computing 69 (2017), 98–117.

[5]

Jaeyoung Choi, Jack Dongarra, Susan Ostrouchov, Antoine Petitet, David W. Walker, and R. Clinton Whaley. 1996. Design and Implementation of the ScaLAPACK LU, QR, and Cholesky Factorization Routines. Scientific Programming 5, 3 (1996), 173–184.

Digital Library

[6]

James Dinan, David Goodell, William Gropp, Rajeev Thakur, and Pavan Balaji. [n. d.]. Efficient Multithreaded Context ID Allocation in MPI. In Recent Advances in the Message Passing Interface. 19th European MPI Users’ Group Meeting (EuroMPI)(Lecture Notes in Computer Science, Vol. 7490). 57–66.

[7]

Markus Geimer, Marc-André Hermanns, Christian Siebert, Felix Wolf, and Brian J. N. Wylie. 2011. Scaling Performance Tool MPI Communicator Management. In Recent Advances in the Message Passing Interface. 18th European MPI Users’ Group Meeting (EuroMPI)(Lecture Notes in Computer Science, Vol. 6960). 178–187.

[8]

William D. Gropp, Torsten Hoefler, Rajeev Thakur, and Jesper Larsson Träff. 2011. Performance expectations and guidelines for MPI derived datatypes: a first analysis. In Recent Advances in Message Passing Interface. 18th European MPI Users’ Group Meeting(Lecture Notes in Computer Science, Vol. 6960). Springer, 150–159.

[9]

Torsten Hoefler, Prabhanjan Kambadur, Richard L. Graham, Galen M. Shipman, and Andrew Lumsdaine. 2007. A Case for Standard Non-blocking Collective Operations. In Recent Advances in Parallel Virtual Machine and Message Passing Interface. 14th European PVM/MPI User’s Group Meeting(Lecture Notes in Computer Science, Vol. 4757). Springer, 125–134.

[10]

Torsten Hoefler, Christian Siebert, and Andrew Lumsdaine. 2009. Group Operation Assembly Language - A Flexible Way to Express Collective Communication. In International Conference on Parallel Processing (ICPP). 574–581.

Digital Library

[11]

Torsten Hoefler and Marc Snir. 2011. Writing Parallel Libraries with MPI – Common Practice, Issues, and Extensions. In Recent Advances in the Message Passing Interface. 18th European MPI Users’ Group Meeting (EuroMPI). 345–355.

Digital Library

[12]

Sascha Hunold, Alexandra Carpen-Amarie, Felix Donatus Lübbe, and Jesper Larsson Träff. 2016. Automatic Verification of Self-Consistent MPI Performance Guidelines. In Euro-Par Parallel Processing(Lecture Notes in Computer Science, Vol. 9833). 433–446.

[13]

Faisal Ghias Mir and Jesper Larsson Träff. 2008. Constructing MPI input-output Datatypes for efficient Transpacking. In Recent Advances in Parallel Virtual Machine and Message Passing Interface. 15th European PVM/MPI Users’ Group Meeting(Lecture Notes in Computer Science, Vol. 5205). Springer, 141–150.

Digital Library

[14]

Faisal Ghias Mir and Jesper Larsson Träff. 2009. Exploiting efficient Transpacking for One-sided Communication and MPI-IO. In Recent Advances in Parallel Virtual Machine and Message Passing Interface. 16th European PVM/MPI Users’ Group Meeting(Lecture Notes in Computer Science, Vol. 5759). Springer, 154–163.

Digital Library

[15]

MPI Forum. 2021. MPI: A Message-Passing Interface Standard. Version 4.0. www.mpi-forum.org.

[16]

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. 2019. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Advances in Neural Information Processing Systems 32. 8024–8035.

[17]

Jack Poulson, Bryan Marker, Robert van de Geijn, Jeff R. Hammond, and Nichols A. Romero. 2013. Elemental: A New Framework for Distributed Memory Dense Matrix Computations. ACM Trans. Math. Software 39, 2 (2013).

Digital Library

[18]

Rajeev Thakur, William Gropp, and Ewing Lusk. 1999. On Implementing MPI-IO Portably and with High Performance. In 6th Workshop on I/O in Parallel and Distributed Systems (IOPADS). 23–32.

[19]

Jesper Larsson Träff. 2016. A Library for Advanced Datatype Programming. In 23rd European MPI Users’ Group Meeting (EuroMPI). ACM, 98–107.

Digital Library

[20]

Jesper Larsson Träff. 2020. Signature datatypes for type correct collective operations, revisited. In 27th European MPI Users’ Group Meeting (EuroMPI/USA). ACM, 1:1–1:8.

Digital Library

[21]

Jesper Larsson Träff, William D. Gropp, and Rajeev Thakur. 2010. Self-consistent MPI Performance Guidelines. IEEE Transactions on Parallel and Distributed Systems 21, 5 (2010), 698–709.

Digital Library

[22]

Jesper Larsson Träff and Sascha Hunold. 2020. Decomposing MPI Collectives for Exploiting Multi-lane Communication. In IEEE International Conference on Cluster Computing (CLUSTER). IEEE Computer Society, 270–280.

[23]

Jesper Larsson Träff, Sascha Hunold, Nikolaus Manes Funk, and Ioannis Vardas. 2023. Uniform Algorithms for Reduce-scatter and (most) other Collectives for MPI. In IEEE International Conference on Cluster Computing (CLUSTER). IEEE Computer Society.

[24]

Jesper Larsson Träff, Sascha Hunold, Guillaume Mercier, and Daniel J. Holmes. 2021. MPI collective communication through a single set of interfaces: A case for orthogonality. Parallel Computing 107 (2021).

[25]

Ioannis Vardas, Sascha Hunold, Jordy I. Ajanohoun, and Jesper Larsson Träff. 2022. mpisee: MPI Profiling for Communication and Communicator Structure. In 27th International Workshop on High-level Parallel Programming Models and Supportive Environments (HIPS), 36th International Parallel and Distributed Processing Symposium (IPDPS). IEEE Computer Society, 520–529.

Cited By

Zhu QDai YPeng JLiang CYang BLiu J(2023)Efficient Approaches to Mitigate Communication Bottlenecks in MPI Communicator Splitting by Type2023 5th International Conference on Frontiers Technology of Information and Computer (ICFTIC)10.1109/ICFTIC59930.2023.10455835(217-222)Online publication date: 17-Nov-2023
https://doi.org/10.1109/ICFTIC59930.2023.10455835
Träff JHunold SVardas IFunk N(2023)Uniform Algorithms for Reduce-scatter and (most) other Collectives for MPI2023 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER52292.2023.00031(284-294)Online publication date: 31-Oct-2023
https://doi.org/10.1109/CLUSTER52292.2023.00031

Index Terms

Library Development with MPI: Attributes, Request Objects, Group Communicator Creation, Local Reductions, and Datatypes
1. Computing methodologies
  1. Parallel computing methodologies
    1. Parallel programming languages
2. Software and its engineering
  1. Software notations and tools
    1. Software libraries and repositories

Recommendations

Implementation and performance analysis of non-blocking collective operations for MPI
SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing

Collective operations and non-blocking point-to-point operations have always been part of MPI. Although non-blocking collective operations are an obvious extension to MPI, there have been no comprehensive studies of this functionality. In this paper we ...
MPI Derived Datatypes: Performance and Portability Issues
EuroMPI '18: Proceedings of the 25th European MPI Users' Group Meeting

This paper addresses performance-portability and overall performance issues when derived datatypes are used with four MPI implementations: Open MPI, MPICH, MVAPICH2, and Intel MPI. These comparisons are particularly relevant today since most vendor ...
The pilot library for novice MPI programmers
PPoPP '10: Proceedings of the 15th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming

The Pilot library is a new method for programming MPI-enabled clusters in C, targeted at novice parallel programmers. Formal elements from Communicating Sequential Processes (CSP) are used to realize a process/channel model of parallel computation that ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

EuroMPI '23: Proceedings of the 30th European MPI Users' Group Meeting

September 2023

123 pages

ISBN:9798400709135

DOI:10.1145/3615318

Copyright © 2023 Owner/Author.

This work is licensed under a Creative Commons Attribution International 4.0 License.

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 21 September 2023

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Funding Sources

Austrian Science Fund

Conference

EUROMPI '23

EUROMPI '23: 30th European MPI Users' Group Meeting

September 11 - 13, 2023

Bristol, United Kingdom

Acceptance Rates

Overall Acceptance Rate 66 of 139 submissions, 47%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
148
Total Downloads

Downloads (Last 12 months)132
Downloads (Last 6 weeks)21

Reflects downloads up to 04 Oct 2024

Other Metrics

View Author Metrics

Citations

Cited By

Zhu QDai YPeng JLiang CYang BLiu J(2023)Efficient Approaches to Mitigate Communication Bottlenecks in MPI Communicator Splitting by Type2023 5th International Conference on Frontiers Technology of Information and Computer (ICFTIC)10.1109/ICFTIC59930.2023.10455835(217-222)Online publication date: 17-Nov-2023
https://doi.org/10.1109/ICFTIC59930.2023.10455835
Träff JHunold SVardas IFunk N(2023)Uniform Algorithms for Reduce-scatter and (most) other Collectives for MPI2023 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER52292.2023.00031(284-294)Online publication date: 31-Oct-2023
https://doi.org/10.1109/CLUSTER52292.2023.00031

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

HTML Format

View this article in HTML Format.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents