Optimizing Amber for Device-to-Device GPU Communication
Authors: 
Samuel Khuvis, Karen Tomko, Scott R. Brozell, Chen-Chun Chen, Hari Subramoni, and Dhabaleswar K. PandaAuthors Info & Claims
Samuel Khuvis, Karen Tomko, Scott R. Brozell, Chen-Chun Chen, Hari Subramoni, and Dhabaleswar K. PandaAuthors Info & ClaimsJuly 2023
Pages 200 - 205
Abstract
Although direct GPU-to-GPU communication has been possible in MPI libraries for over a decade, the limited availability of compatible hardware at academic HPC centers has discouraged the development of algorithms in scientific applications that take advantage of this capability. In this paper, we take Amber, a molecular dynamics code used to simulate proteins and nucleic acids, as a test case. We demonstrate the modifications necessary to implement GPU-to-GPU communication. Compared to the previous implementation, these modifications show an average of approximately 36% improvement in performance overall and 84% for the important explicit solvent subset of the benchmarks.
References
[1]
M. P. Allen and D. J. Tildesley. 1987. Computer Simulation of Liquids. Clarendon Press, Oxford.
[2]
D.A. Case, H.M. Aktulga, K. Belfon, I.Y. Ben-Shalom, J.T. Berryman, S.R. Brozell, D.S. Cerutti, T.E. Cheatham, III, G.A. Cisneros, V.W.D. Cruzeiro, T.A. Darden, R.E. Duke, G. Giambasu, M.K. Gilson, H. Gohlke, A.W. Goetz, R. Harris, S. Izadi, S.A. Izmailov, K. Kasavajhala, M.C. Kaymak, E. King, A. Kovalenko, T. Kurtzman, T.S. Lee, S. LeGrand, P. Li, C. Lin, J. Liu, T. Luchko, R. Luo, M. Machado, V. Man, M. Manathunga, K.M. Merz, Y. Miao, O. Mikhailovskii, G. Monard, H. Nguyen, K.A. O’Hearn, A. Onufriev, F. Pan, S. Pantano, R. Qi, A. Rahnamoun, D.R. Roe, A. Roitberg, C. Sagui, S. Schott-Verdugo, A. Shajan, J. Shen, C.L. Simmerling, N.R. Skrynnikov, J. Smith, J. Swails, R.C. Walker, J. Wang, J. Wang, H. Wei, R.M. Wolf, X. Wu, Y. Xiong, Y. Xue, D.M. York, S. Zhao, and P.A. Kollman. 2020. Amber 2020. https://ambermd.org. University of California, San Francisco.
[3]
Ohio Supercomputer Center. 1987. Ohio Supercomputer Center. http://osc.edu/ark:/19495/f5s1ph73
[4]
Amber Project Contributors. 2023. GPU overview and brief history. Amber Project. Retrieved February 27, 2023 from https://ambermd.org/GPUSupport.php
[5]
Paul Stewart Crozier, Gilad Shainer, Ali Ayoub, Pak Lui, Tong Liu, Christian Robert Trott, and Greg Scantlen. 2011. The Development of Mellanox - NVIDIA GPUDirect over InfiniBand - a New Model for GPU to GPU Communications. (January 2011). https://www.osti.gov/biblio/1120826
[6]
GROMACS development team. 2022. Highlights. GROMACS Project. Retrieved February 27, 2023 from https://manual.gromacs.org/documentation/2022/release-notes/2022/major/highlights.html
[7]
Andreas W. Götz, Mark J. Williamson, Dong Xu, Duncan Poole, Scott Le Grand, and Ross C. Walker. 2012. Routine Microsecond Molecular Dynamics Simulations with AMBER on GPUs. 1. Generalized Born. Journal of Chemical Theory and Computation 8, 5 (2012), 1542–1555. https://doi.org/10.1021/ct200909j arXiv:https://doi.org/10.1021/ct200909jPMID: 22582031.
[8]
ARM Ltd.2022. ARM MAP. https://www.linaroforge.com
[9]
Dhabaleswar Kumar Panda, Hari Subramoni, Ching-Hsiang Chu, and Mohammadreza Bayatpour. 2021. The MVAPICH project: Transforming research into high-performance MPI library for HPC community. Journal of Computational Science 52 (2021), 101208. https://doi.org/10.1016/j.jocs.2020.101208 Case Studies in Translational Computer Science.
[10]
Daniel R. Roe and Bernard R. Brooks. 2020. A Protocol for Preparing Explicitly Solvated Systems for Stable Molecular Dynamics Simulations. The Journal of Chemical Physics 153 (2020), 054123. https://doi.org/10.1063/5.0013849
[11]
Romelia Salomon-Ferrer, Andreas W. Götz, Duncan Poole, Scott Le Grand, and Ross C. Walker. 2013. Routine Microsecond Molecular Dynamics Simulations with AMBER on GPUs. 2. Explicit Solvent Particle Mesh Ewald. Journal of Chemical Theory and Computation 9, 9 (2013), 3878–3888. https://doi.org/10.1021/ct400314y arXiv:https://doi.org/10.1021/ct400314yPMID: 26592383.
[12]
Tamar Schlick, Stephanie Portillo-Ledesma, Christopher G. Myers, Lauren Beljak, Justin Chen, Sami Dakhel, Daniel Darling, Sayak Ghosh, Joseph Hall, Mikaeel Jan, Emily Liang, Sera Saju, Mackenzie Vohr, Chris Wu, Yifan Xu, and Eva Xue. 2021. Biomolecular Modeling and Simulation: A Prospering Multidisciplinary Field. Annual Review of Biophysics 50, 1 (2021), 267–301. https://doi.org/10.1146/annurev-biophys-091720-102019 arXiv:https://doi.org/10.1146/annurev-biophys-091720-102019PMID: 33606945.
[13]
Hsu-Chun Tsai, Tai-Sung Lee, Abir Ganguly, Timothy J. Giese, Maximilian CCJC Ebert, Paul Labute, Kenneth M. Jr. Merz, and Darrin M. York. 2023. AMBER Free Energy Tools: A New Framework for the Design of Optimized Alchemical Transformation Pathways. Journal of Chemical Theory and Computation 19, 2 (2023), 640–658. https://doi.org/10.1021/acs.jctc.2c00725 arXiv:https://doi.org/10.1021/acs.jctc.2c00725PMID: 36622640.
[14]
Hao Wang, Sreeram Potluri, Miao Luo, Ashish Kumar Singh, Sayantan Sur, and Dhabaleswar K. Panda. 2011. MVAPICH2-GPU: optimized GPU to GPU communication for InfiniBand clusters.Comput Sci Res Dev 26 (2011), 257–266. https://doi.org/10.1007/s00450-011-0171-3
Index Terms
- Optimizing Amber for Device-to-Device GPU Communication
Recommendations
Exploiting computing power of Xeon and Intel Xeon Phi for a molecular dynamics application
HPC '15: Proceedings of the Symposium on High Performance ComputingMolecular Dynamics (MD) is a computational technique with applicability in fields as diverse as material science, biomolecules and chemical physics. Assisted Model Building with Energy Refinement (AMBER) is an MD package and it uses Message Passing ...
Consensus in silico computational modelling of the p22phox subunit of the NADPH oxidase
The p22^p^h^o^x protein is an essential subunit of the cytochrome b"5"5"8 of the NADPH oxidase (Nox) complex which by generating reactive oxygen species (ROS) plays important role in regulating cellular function. p22^p^h^o^x stabilises the Nox enzyme, ...
Analyzing GPU-controlled communication with dynamic parallelism in terms of performance and energy
Intra-GPU synchronization is a problem for GPU controlled communication.Options, based on dynamic parallelism provide on-device synchronization.GPU controlled communication have a lower performance than CPU assisted approaches.Relieving the CPU from the ...
Comments
Information & Contributors
Information
Published In
July 2023
519 pages
ISBN:9781450399852
DOI:10.1145/3569951
Copyright © 2023 ACM.
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].
Sponsors
Publisher
Association for Computing Machinery
New York, NY, United States
Publication History
Published: 10 September 2023
Check for updates
Author Tags
Qualifiers
- Short-paper
- Research
- Refereed limited
Funding Sources
Conference
PEARC '23: Practice and Experience in Advanced Research Computing
July 23 - 27, 2023
OR, Portland, USA
Acceptance Rates
Overall Acceptance Rate 133 of 202 submissions, 66%
Upcoming Conference
Contributors
Other Metrics
Bibliometrics & Citations
Bibliometrics
Article Metrics
- 0Total Citations
- 84Total Downloads
- Downloads (Last 12 months)84
- Downloads (Last 6 weeks)3
Other Metrics
Citations
View Options
Get Access
Login options
Check if you have access through your login credentials or your institution to get full access on this article.
Sign inFull Access
View options
View or Download as a PDF file.
PDFeReader
View online with eReader.
eReaderHTML Format
View this article in HTML Format.
HTML Format