Abstract
MPICH2 provides a layered architecture to achieve both portability and performance. For implementations of MPI-2 over InfiniBand, it provides the flexibility for researchers at the RDMA channel, CH3 or ADI3 layer. In this paper we analyze the performance and complexity trade-offs associated with implementations at these layers. We describe our designs and implementations, as well as optimizations at each layer. To show the performance impacts of these design schemes and optimizations, we evaluate our implementations with different micro-benchmarks, HPCC and NAS test suite. Our experiments show that although the ADI3 layers adds complexity in implementation, the benefits achieved through optimizations justify moving to the ADI layer to extract the best performance.
This research is supported in part by Department of Energy’s Grant #DE-FC02-01ER25506, National Science Foundation’s grants #CNS-0204429, and #CUR-0311542, and a grant from Intel.
This is a preview of subscription content, log in via an institution to check access.
Preview
Unable to display preview. Download preview PDF.
Similar content being viewed by others
References
Bailey, D.H., Barszcz, E., Dagum, L., Simon, H.D.: NAS Parallel Benchmark Results. Technical Report 94-006, RNR (1994)
HPC Challenge Benchmark, http://icl.cs.utk.edu/hpcc/
Grabner, R., Mietke, F., Rehm, W.: An MPICH2 Channel Device Implementation over VAPI on InfiniBand. In: Proceedings of the International Parallel and Distributed Processing Symposium (2004)
Huang, W., Santhanaraman, G., Jin, H.W., Panda, D.K.: Scheduling of MPI-2 One Sided Operations over InfiniBand. In: Workshop On Communication Architecture on Clusters (CAC), in conjunction with IPDPS 2005 (April 2005)
InfiniBand Trade Association. InfiniBand Architecture Specification, Release 1.2
Network Based Computing Laboratory, http://nowlab.cis.ohio-state.edu/
Liu, J., Jiang, W., Jin, H.W., Panda, D.K., Gropp, W., Thakur, R.: High Performance MPI-2 One-Sided Communication over InfiniBand. In: International Symposium on Cluster Computing and the Grid (CCGrid 2004) (April 2004)
Liu, J., Jiang, W., Wyckoff, P., Panda, D.K., Ashton, D., Buntinas, D., Gropp, W., Toonen, B.: Design and Implementation of MPICH2 over InfiniBand with RDMA Support. In: Proceedings of the International Parallel and Distributed Processing Symposium (2004)
Message Passing Interface Forum. MPI-2: A Message Passing Interface Standard. High Performance Computing Applications 12(1–2), 1–299 (1998)
Santhanaraman, G., Wu, J., Panda, D.K.: Zero-Copy MPI Derived Datatype Communication over InfiniBand. In: EuroPVM-MPI 2004 (September 2004)
Snir, M., Otto, S., Huss-Lederman, S., Walker, D., Dongarra, J.: MPI–The Complete Reference, 2nd edn. The MPI-1 Core, vol. 1. The MIT Press, Cambridge (1998)
Tezuka, H., O’Carroll, F., Hori, A., Ishikawa, Y.: Pin-down cache: A virtual memory management technique for zero-copy communication. In: Proceedings of the 12th International Parallel Processing Symposium (1998)
Wu, J., Wyckoff, P., Panda, D.K.: High Performance Implementation of MPI Datatype Communication over InfiniBand. In: Proceedings of the International Parallel and Distributed Processing Symposium (2004)
Author information
Authors and Affiliations
Editor information
Editors and Affiliations
Rights and permissions
Copyright information
© 2005 Springer-Verlag Berlin Heidelberg
About this paper
Cite this paper
Huang, W., Santhanaraman, G., Jin, HW., Panda, D.K. (2005). Design Alternatives and Performance Trade-Offs for Implementing MPI-2 over InfiniBand. In: Di Martino, B., Kranzlmüller, D., Dongarra, J. (eds) Recent Advances in Parallel Virtual Machine and Message Passing Interface. EuroPVM/MPI 2005. Lecture Notes in Computer Science, vol 3666. Springer, Berlin, Heidelberg. https://doi.org/10.1007/11557265_27
Download citation
DOI: https://doi.org/10.1007/11557265_27
Publisher Name: Springer, Berlin, Heidelberg
Print ISBN: 978-3-540-29009-4
Online ISBN: 978-3-540-31943-6
eBook Packages: Computer ScienceComputer Science (R0)