research-article

Public Access

Exploiting Process Imbalance to Improve MPI Collective Operations in Hierarchical Systems

Authors:

Benjamin S. Parsons,

Vijay S. PaiAuthors Info & Claims

ICS '15: Proceedings of the 29th ACM on International Conference on Supercomputing

Pages 57 - 66

https://doi.org/10.1145/2751205.2751242

Published: 08 June 2015 Publication History

Abstract

This work improves the performance of MPI collective communication operations in the presence of imbalanced process arrival times. High performance collective communications are crucial for the performance and scalability of applications, and imbalanced process arrival times are common in these applications. A micro-benchmark is used to investigate the nature of process imbalance with perfectly balanced workloads, and understand the nature of inter- versus intra-node imbalance. These insights are then used to develop imbalance-tolerant reduction, broadcast, and all-to-all algorithms, which minimize the synchronization delay observed by early arriving processes.

These algorithms have been implemented and tested on a Cray XE6 using up to 32k cores with varying buffer sizes and levels of imbalance. Results show speedups over MPICH averaging 18.9x for reduce, 5.3x for broadcast, and 6.9x for all-to-all in the presence of high, but not unreasonable, imbalance.

References

[1]

MPICH, a high-performance and portable implementation of the MPI. http://www.mpich.org/. Accessed 1/2014.

[2]

mpiP: Lightweight, scalable MPI profiling. http://mpip.sourceforge.net/. Accessed 4/21/2014.

[3]

NAS parallel benchmarks. https://www.nas.nasa.gov/publications/npb.html. Accessed 4/21/2014.

[4]

P. Balaji, D. Buntinas, D. Goodell, W. Gropp, S. Kumar, E. Lusk, R. Thakur, and J. Träff. MPI on a million processors. Lecture Notes in Computer Science. Springer Berlin Heidelberg, 2009.

Digital Library

[5]

F. Cristian. Probabilistic clock synchronization. Distributed Computing, 3(3):146--158, 1989.

Digital Library

[6]

A. Faraj, P. Patarasuk, and X. Yuan. A study of process arrival patterns for MPI collective operations. International Journal of Parallel Programming, 36(6):543--570, 2008.

Digital Library

[7]

A. Faraj, X. Yuan, and D. Lowenthal. STAR-MPI: Self tuned adaptive routines for MPI collective operations. In Proceedings of the 20th Annual International Conference on Supercomputing, ICS '06, pages 199--208, New York, NY, USA, 2006. ACM.

Digital Library

[8]

R. Graham, M. Venkata, J. Ladd, P. Shamis, I. Rabinovitz, V. Filipov, and G. Shainer. Cheetah: A framework for scalable hierarchical collective operations. In Cluster, Cloud and Grid Computing, 2011 11th IEEE/ACM Int. Symp. on, 2011.

Digital Library

[9]

R. Kumar, A. Mamidala, and D. Panda. Scaling alltoall collective on multi-core systems. In Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, pages 1--8, April 2008.

[10]

A. R. Mamidala, R. Kumar, D. De, and D. K. Panda. MPI collectives on modern multicore clusters: Performance optimizations and communication characteristics. In Proc. 2008 8th IEEE Int. Symp. Cluster Computing and Grid, 2008.

Digital Library

[11]

A. R. Mamidala, J. Liu, and D. K. Panda. Efficient barrier and allreduce infiniband clusters using hardware multicast and adaptive algorithms. In In Proceedings of Cluster Computing, 2004.

Digital Library

[12]

P. J. Mucci, S. Browne, C. Deane, and G. Ho. PAPI: A portable interface to hardware performance counters. In In Proceedings of the Department of Defense HPCMP Users Group Conference, pages 7--10, 1999.

[13]

B. S. Parsons and V. S. Pai. Accelerating MPI collective communications through hierarchical algorithms without sacrificing inter-node communication flexibility. In Parallel and Distributed Processing Symposium, IEEE 28th International, May 2014.

Digital Library

[14]

P. Patarasuk and X. Yuan. Efficient MPI bcast across different process arrival patterns. In Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, pages 1--11, April 2008.

[15]

Y. Qian and A. Afsahi. Process arrival pattern and shared memory aware alltoall on Infiniband. In Proceedings of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface, pages 250--260, Berlin, Heidelberg, 2009. Springer-Verlag.

Digital Library

[16]

Y. Qian and A. Afsahi. Process arrival pattern aware alltoall and allgather on Infiniband clusters. International Journal of Parallel Programming, 39(4):473--493, 2011.

[17]

S. Sistare, R. vandeVaart, and E. Loh. Optimization of MPI collectives on clusters of large-scale SMP's. In Proc. of the 1999 ACM/IEEE Conf. on Supercomputing. ACM, 1999.

Digital Library

[18]

J. S. Vetter and F. Mueller. Communication characteristics of large-scale scientific applications for contemporary cluster architectures. J. Parallel Distrib. Comput., 63(9):853--865, Sept. 2003.

Digital Library

Cited By

Beni MCosenza BHunold S(2024)MPI Collective Algorithm Selection in the Presence of Process Arrival Patterns2024 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER59578.2024.00017(108-119)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTER59578.2024.00017
Luo XWu WBosilca GPei YCao QPatinyasakdikul TZhong DDongarra J(2020)HAN: a Hierarchical AutotuNed Collective Communication Framework2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00013(23-34)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00013
Margolin ABarak A(2020)Tree‐based fault‐tolerant collective operations for MPIConcurrency and Computation: Practice and Experience10.1002/cpe.582633:14Online publication date: 15-Jun-2020
https://doi.org/10.1002/cpe.5826
Show More Cited By

Index Terms

Exploiting Process Imbalance to Improve MPI Collective Operations in Hierarchical Systems
1. Software and its engineering

Recommendations

A study of process arrival patterns for MPI collective operations
ICS '07: Proceedings of the 21st annual international conference on Supercomputing

Process arrival pattern, which denotes the timing when different processes arrive at an MPI collective operation, can have a significant impact on the performance of the operation. In this work, we characterize the process arrival patterns in a set of ...
Kernel-Assisted MPI Collective Communication among Many-core Clusters
CCGRID '12: Proceedings of the 2012 12th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing (ccgrid 2012)

Architectural hierarchies and hardware complexity brought by multicore or many-core Clusters, greatly challenge MPI applications' performance in two ways: performance efficiency and cross-platform portability. The cross-platform portability assumption, '...
Optimization of MPI collective communication on BlueGene/L systems
ICS '05: Proceedings of the 19th annual international conference on Supercomputing

BlueGene/L is currently the world's fastest supercomputer. It consists of a large number of low power dual-processor compute nodes interconnected by high speed torus and collective networks, Because compute nodes do not have shared memory, MPI is the ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

ICS '15: Proceedings of the 29th ACM on International Conference on Supercomputing

June 2015

446 pages

ISBN:9781450335591

DOI:10.1145/2751205

General Chair:
Laxmi N. Bhuyan
University of California, Riverside
,
Program Chairs:
Fred Chong
University of California, Santa Barbara
,
Vivek Sarkar
Rice University

Copyright © 2015 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2015

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

NSF
DOE

Conference

ICS'15

Sponsor:

SIGARCH

ICS'15: 2015 International Conference on Supercomputing

June 8 - 11, 2015

California, Newport Beach, USA

Acceptance Rates

ICS '15 Paper Acceptance Rate 40 of 160 submissions, 25%;

Overall Acceptance Rate 629 of 2,180 submissions, 29%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
333
Total Downloads

Downloads (Last 12 months)65
Downloads (Last 6 weeks)19

Reflects downloads up to 12 Feb 2025

Other Metrics

View Author Metrics

Citations

Cited By

Beni MCosenza BHunold S(2024)MPI Collective Algorithm Selection in the Presence of Process Arrival Patterns2024 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER59578.2024.00017(108-119)Online publication date: 24-Sep-2024
https://doi.org/10.1109/CLUSTER59578.2024.00017
Luo XWu WBosilca GPei YCao QPatinyasakdikul TZhong DDongarra J(2020)HAN: a Hierarchical AutotuNed Collective Communication Framework2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00013(23-34)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00013
Margolin ABarak A(2020)Tree‐based fault‐tolerant collective operations for MPIConcurrency and Computation: Practice and Experience10.1002/cpe.582633:14Online publication date: 15-Jun-2020
https://doi.org/10.1002/cpe.5826
Utrera GGil MMartorell X(2017)Analyzing the impact of communication imbalance in high‐speed networksConcurrency and Computation: Practice and Experience10.1002/cpe.439430:7Online publication date: 20-Dec-2017
https://doi.org/10.1002/cpe.4394

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Figures

Tables

Media

View Table of Conten