research-article

Unifying UPC and MPI runtimes: experience with MVAPICH

Authors:

Dhabaleswar K. PandaAuthors Info & Claims

PGAS '10: Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model

Article No.: 5, Pages 1 - 10

https://doi.org/10.1145/2020373.2020378

Published: 12 October 2010 Publication History

Abstract

Unified Parallel C (UPC) is an emerging parallel programming language that is based on a shared memory paradigm. MPI has been a widely ported and dominant parallel programming model for the past couple of decades. Real-life scientific applications require a lot of investment by domain scientists. Many scientists choose the MPI programming model as it is considered low-risk. It is unlikely that entire applications will be re-written using the emerging UPC language (or PGAS paradigm) in the near future. It is more likely that parts of these applications will be converted to newer models. This requires that underlying implementation of system software be able to support both UPC and MPI simultaneously. Unfortunately, the current state-of-the-art of UPC and MPI interoperability leaves much to be desired both in terms of performance and ease-of-use.

In this paper, we propose "Integrated Native Communication Runtime" (INCR) for MPI and UPC communications on InfiniBand clusters. Our library is capable of supporting both UPC and MPI communications simultaneously. This runtime is based on the widely used MVAPICH (MPI over InfiniBand) Aptus runtime, which is known to scale to tens-of-thousands of cores. Our evaluation reveals that INCR is able to deliver equal or better performance compared to the existing UPC runtime - GASNet on InfiniBand verbs. We observe that with UPC NAS benchmarks CG and MG (class B) at 128 processes, we outperform current GASNet implementation by 10% and 23%, respectively.

References

[1]

Parallel Environment for AIX 5L MPI Programming Guide. (v4.1.1): Using MPI and LAPI in the same program.

[2]

Berkeley UPC User's Guide: Mixing UPC and C/C++/MPI/FORTRA. http://upc.lbl.gov/docs/user/interoperability.shtml.

[3]

C. Barton, C. Casçaval, G. Almási, Y. Zheng, M. Farreras, S. Chatterjee, and J. N. Amaral. Shared memory programming for large scale machines. In PLDI '06: Proceedings of the 2006 ACM SIGPLAN conference on Programming language design and implementation, pages 108--117, New York, NY, USA, 2006. ACM. ISBN 1-59593-320-4.

Digital Library

[4]

C. Bell and D. Bonachea. A New DMA Registration Strategy for Pinning-Based High Performance Networks. In Workshop on Communication Architecture for Clusters (CAC), in conjunction with Int'l Symposium on Parallel and Distributed Computing (IPDPS), 2003.

Digital Library

[5]

C. Bell and D. Bonachea and R. Nishtala and K. Yelick. Optimizing Bandwidth Limited Problems Using One-Sided Communication and Overlap. In Int'l Symposium on Parallel and Distributed Computing (IPDPS), 2006.

Digital Library

[6]

B. Chamberlain, D. Callahan, and H. Zima. Parallel Programmability and the Chapel Language. Int. J. High Perform. Comput. Appl., 21(3):291--312, 2007. ISSN 1094-3420.

Digital Library

[7]

P. Charles, C. Grothoff, V. Saraswat, C. Donawa, A. Kielstra, K. Ebcioglu, C. von Praun, and V. Sarkar. X10: an object-oriented approach to non-uniform cluster computing. In OOPSLA '05: Proceedings of the 20th annual ACM SIGPLAN conference on Object-oriented programming, systems, languages, and applications, pages 519--538, New York, NY, USA, 2005. ACM. ISBN 1-59593-031-0.

Digital Library

[8]

D. Bonachea and J. Duell. Problems with using MPI 1.1 and 2.0 as Compilation Targets for Parallel Language Implementations. Intl. J. High Performance Computing and Networking, 1(1/2/3):91--99, 2004.

Digital Library

[9]

D. Buntinas and W. Gropp. Designing a Common Communication Subsystem. In Euro PVM/MPI Conference, 2005.

Digital Library

[10]

Editor: Dan Bonachea. GASNet specification v1.1. Technical Report UCB/CSD-02-1207, U. C. Berkeley, 2008.

Digital Library

[11]

Environmental Molecular Sciences Laboratory and Pacific Northwest National Laboratory. The GA Toolkit. http://www.emsl.pnl.gov/docs/global/.

[12]

Environmental Molecular Sciences Laboratory, Pacific Northwest National Laboratory. Aggregate Remote Memory Copy Interface (ARMCI). http://www.emsl.pnl.gov/docs/parsoft/armci/.

[13]

IBM. Message Passing Interface and Low-Level Application Programming Interface (LAPI). http://www-03.ibm.com/systems/software/parallel/index.html.

[14]

InfiniBand Trade Association. InfiniBand Industry Standard Specification. http://www.infinibandta.org/.

[15]

M. Koop, T. Jones, and D. K. Panda. Reducing Connection Memory Requirements of MPI for InfiniBand Clusters: A Message Coalescing Approach. In 7th IEEE Int'l Symposium on Cluster Computing and the Grid (CCGrid07), May 2007.

Digital Library

[16]

M. Koop, S. Sur, Q. Gao, and D. K. Panda. High Performance MPI Design using Unreliable Datagram for Ultra-Scale InfiniBand Clusters. In 21st ACM International Conference on Supercomputing (ICS07), June 2007.

Digital Library

[17]

M. Koop, T. Jones, and D. K. Panda. MVAPICH-Aptus: Scalable High-Performance Multi-Transport MPI over InfiniBand. In IEEE Int'l Parallel and Distributed Processing Symposium (IPDPS 2008), April 2008.

[18]

M. Koop, J. Sridhar, and D. K. Panda. Scalable MPI Design over InfiniBand using eXtended Reliable Connection. In IEEE Int'l Conference on Cluster Computing (Cluster 2008), September 2008.

[19]

Lawrence Berkeley National Laboratory and University of California at Berkeley. Berkeley UPC - Unified Parallel C. http://upc.lbl.gov/.

[20]

J. Liu, J. Wu, and D. K. Panda. High Performance RDMA-Based MPI Implementation over InfiniBand. In In 17th Annual ACM International Conference on Supercomputing (ICS), 2003.

Digital Library

[21]

D. Loveman. High performance Fortran. Parallel Distributed Technology: Systems Applications, IEEE, 1(1):25--42, feb. 1993. ISSN 1063-6552.

Digital Library

[22]

National Aeronautics and Space Administration (NASA). NAS Parallel Benchmarks. http://www.nas.nasa.gov/Resources/Software/npb.html.

[23]

Network-Based Computing Laboratory. MVAPICH: MPI over InfiniBand, 10GigE/iWARP and RoCE. http://mvapich.cse.ohio-state.edu/.

[24]

R. Nishtala and P. Hargrove and D. Bonachea and K. Yelick. Scaling Communication-Intensive Applications on BlueGene/P Using One-Sided Communication and Overlap. In Int'l Symposium on Parallel and Distributed Computing (IPDPS), 2009.

Digital Library

[25]

Remote Memory Access Working Group. MPI3 RMA Wiki. https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage.

[26]

S. Sur, L. Chai, H.-W. Jin, and D. K. Panda. Shared Receive Queue Based Scalable MPI Design for InfiniBand Clusters. In IEEE Int'l Parallel and Distributed Processing Symposium (IPDPS 2006), April 2006.

Digital Library

[27]

UPC Consortium. UPC Language Specifications, v1.2. Technical Report LBNL-59208, Lawrence Berkeley National Lab, 2005.

Cited By

Wu QBeard JEkanayake AGerstlauer AJohn L(2021)Virtual-Link: A Scalable Multi-Producer Multi-Consumer Message Queue Architecture for Cross-Core Communication2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS49936.2021.00027(182-191)Online publication date: May-2021
https://doi.org/10.1109/IPDPS49936.2021.00027
Coti CMalony A(2021)DiPOSH: A portable OpenSHMEM implementation for short API‐to‐network pathConcurrency and Computation: Practice and Experience10.1002/cpe.617933:11Online publication date: 4-Feb-2021
https://doi.org/10.1002/cpe.6179
Khuvis STomko KHashmi JPanda D(2020)Exploring Hybrid MPI+Kokkos Tasks Programming Model2020 IEEE/ACM 3rd Annual Parallel Applications Workshop: Alternatives To MPI+X (PAW-ATM)10.1109/PAWATM51920.2020.00011(66-73)Online publication date: Nov-2020
https://doi.org/10.1109/PAWATM51920.2020.00011
Show More Cited By

Index Terms

Unifying UPC and MPI runtimes: experience with MVAPICH
1. Software and its engineering
  1. Software creation and management
    1. Software development techniques
  2. Software notations and tools
    1. Compilers
      1. Runtime environments
    2. General programming languages
      1. Language features

Recommendations

MPI + MPI: a new hybrid approach to parallel programming with MPI plus shared memory

Hybrid parallel programming with the message passing interface (MPI) for internode communication in conjunction with a shared-memory programming model to manage intranode parallelism has become a dominant approach to scalable parallel programming. While ...
A UPC Runtime System Based on MPI and POSIX Threads
PDP '06: Proceedings of the 14th Euromicro International Conference on Parallel, Distributed, and Network-Based Processing

MuPC is a portable runtime system for Unified Parallel C (UPC). A modified version of the EDG C/C++ front end translates the user's UPC program into C and turns UPC-specific language features into calls to MuPC runtime functions. MuPC implements each ...
Performance Evaluation of MPI, UPC and OpenMP on Multicore Architectures
Proceedings of the 16th European PVM/MPI Users' Group Meeting on Recent Advances in Parallel Virtual Machine and Message Passing Interface

The current trend to multicore architectures underscores the need of parallelism. While new languages and alternatives for supporting more efficiently these systems are proposed, MPI faces this new challenge. Therefore, up-to-date performance ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PGAS '10: Proceedings of the Fourth Conference on Partitioned Global Address Space Programming Model

October 2010

134 pages

ISBN:9781450304610

DOI:10.1145/2020373

General Chair:
José E. Moreira
IBM T.J. Watson Research Center
,
Program Chairs:
Costin Iancu
Lawrence Berkeley Laboratory
,
Vijay Saraswat
IBM T.J. Watson Research Center

Copyright © 2010 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 October 2010

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

Conference

PGAS '10

PGAS '10: Fourth Conference on Partitioned Global Address Space Programming Model

October 12 - 15, 2010

New York, New York, USA

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

38
Total Citations
View Citations
145
Total Downloads

Downloads (Last 12 months)2
Downloads (Last 6 weeks)0

Reflects downloads up to 13 Nov 2024

Other Metrics

View Author Metrics

Citations

Cited By

Wu QBeard JEkanayake AGerstlauer AJohn L(2021)Virtual-Link: A Scalable Multi-Producer Multi-Consumer Message Queue Architecture for Cross-Core Communication2021 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS49936.2021.00027(182-191)Online publication date: May-2021
https://doi.org/10.1109/IPDPS49936.2021.00027
Coti CMalony A(2021)DiPOSH: A portable OpenSHMEM implementation for short API‐to‐network pathConcurrency and Computation: Practice and Experience10.1002/cpe.617933:11Online publication date: 4-Feb-2021
https://doi.org/10.1002/cpe.6179
Khuvis STomko KHashmi JPanda D(2020)Exploring Hybrid MPI+Kokkos Tasks Programming Model2020 IEEE/ACM 3rd Annual Parallel Applications Workshop: Alternatives To MPI+X (PAW-ATM)10.1109/PAWATM51920.2020.00011(66-73)Online publication date: Nov-2020
https://doi.org/10.1109/PAWATM51920.2020.00011
Jansson N(2020)A Hybrid MPI+PGAS Approach to Improve Strong Scalability Limits of Finite Element Solvers2020 IEEE International Conference on Cluster Computing (CLUSTER)10.1109/CLUSTER49012.2020.00041(303-313)Online publication date: Sep-2020
https://doi.org/10.1109/CLUSTER49012.2020.00041
Lu FFang TZhang ZLi SChen JAn HHan W(2019)Improving the Performance of MongoDB with RDMA2019 IEEE 21st International Conference on High Performance Computing and Communications; IEEE 17th International Conference on Smart City; IEEE 5th International Conference on Data Science and Systems (HPCC/SmartCity/DSS)10.1109/HPCC/SmartCity/DSS.2019.00144(1004-1010)Online publication date: Aug-2019
https://doi.org/10.1109/HPCC/SmartCity/DSS.2019.00144
Li MWen KLin HJin XWu ZAn HChi M(2019)Improving the Performance of Distributed MXNet with RDMAInternational Journal of Parallel Programming10.1007/s10766-018-00623-w47:3(467-480)Online publication date: 1-Jun-2019
https://dl.acm.org/doi/10.1007/s10766-018-00623-w
Pathak APandey MRautaray S(2019)Approaches of enhancing interoperations among high performance computing and big data analytics via augmentationCluster Computing10.1007/s10586-019-02960-yOnline publication date: 3-Aug-2019
https://doi.org/10.1007/s10586-019-02960-y
Nakao MMurai HBoku TSato MEndo TYokokawa MHanawa TTatebe O(2018)Linkage of XcalableMP and Python languages for high productivity on HPC cluster systemProceedings of Workshops of HPC Asia10.1145/3176364.3176369(39-47)Online publication date: 31-Jan-2018
https://dl.acm.org/doi/10.1145/3176364.3176369
Biswas RLu XPanda D(2018)Accelerating TensorFlow with Adaptive RDMA-Based gRPC2018 IEEE 25th International Conference on High Performance Computing (HiPC)10.1109/HiPC.2018.00010(2-11)Online publication date: Dec-2018
https://doi.org/10.1109/HiPC.2018.00010
Hamidouche KZhang JPanda DTomko KMorris K(2016)OpenSHMEM nonblocking data movement operations with MVAPICH2-XProceedings of the First Workshop on PGAS Applications10.5555/3019040.3019042(9-16)Online publication date: 13-Nov-2016
https://dl.acm.org/doi/10.5555/3019040.3019042
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents