research-article

A distributed memory out-of-core method on HPC clusters and its application to quantum chemistry applications

Authors:

Christopher S. Simmons,

Karl W. SchulzAuthors Info & Claims

XSEDE '12: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond

Article No.: 1, Pages 1 - 7

https://doi.org/10.1145/2335755.2335785

Published: 16 July 2012 Publication History

Abstract

Out-of-core methods, which repeatedly offload data to disk in order to overcome local on-node memory constraints are encountered in a range of scientific computing disciplines, including quantum chemistry. Unfortunately, these methods do not often map nicely onto global parallel file systems employed on modern HPC clusters and can overwhelm even the most capable of file systems causing unacceptably low application performance (while also degrading I/O performance for all system users). To address this bottleneck and explore more efficient use of HPC clusters for a quantum chemistry application, CFOUR, a new MPI-based utility has been developed to support out-of-core methods on distributed memory systems. This MPI Ocore utility leverages the high-speed interconnect available on HPC clusters to offload and retrieve out-of-core records to one or more remote memory storage pools, avoiding excessive I/O transactions on local or global file systems. In this paper, we present an overview of the Ocore implementation, it's direct application within a large quantum chemistry application, and micro-benchmark and application performance results from an HPC cluster interconnected with InfiniBand.

References

[1]

J. Almlöf and P. R. Taylor. General contraction of gaussian basis sets. i. atomic natural orbitals for first-and second-row atoms. The Journal of Chemical Physics, 86(7):4070--4077, 1987.

[2]

F. S. Foundation. GNU Library General Public License, Version 2.0. http://www.gnu.org/licenses/old-licenses/lgpl-2.0-standalone.html.

[3]

M. E. Harding, T. Metzroth, J. Gauss, and A. A. Auer. Parallel calculation of ccsd and ccsd(t) analytic first and second derivatives. Journal of Chemical Theory and Computation, 4(1):64--74, 2008.

[4]

T. Ichino, J. Gauss, and J. Stanton. Quasidiabatic states described by coupled-cluster theory. The Journal of Chemical Physics, Jan 2009.

[5]

M. Isenburg and S. Gumhold. Out-of-core compression for gigantic polygon meshes. ACM Trans. Graph., 22:935--942, July 2003.

Digital Library

[6]

K. Klein, J. Stanton, J. Zhou, and T. Yacovitch. Vibronic structure of the formyloxyl radical (hco2) via slow photoelectron velocity-map imaging spectroscopy and model hamiltonian calculations. The Journal of Physical Chemistry, Jan 2009.

[7]

H. Köppel and W. Domcke.... Multimode molecular dynamics beyond the born-oppenheimer approximation. Advances in chemical physics, Jan 1984.

[8]

H. Krishnan, S. Krishnamoorthy, G. Baumgartner, C. chung Lam, J. Ramanujam, and. D. E. Bernholdt. Data locality optimization for synthesis of efficient out-of-core algorithms. In In Proc. of the Intl. Conf. on High Performance Computing, pages 406--417. Springer Verlag, 2003.

[9]

V. K. Kundeti, S. Rajasekaran, H. Dinh, M. Vaughn, and V. Thapar. Efficient parallel and out of core algorithms for constructing large bi-directed de Bruijn graphs. BMC Bioinformatics, 11:560, 2010.

[10]

libGRVY. Toolkit for HPC Application Development. https://red.ices.utexas.edu/projects/software/wiki/GRVY.

[11]

J. Liu, B. Chandrasekaran, W. Yu, J. Wu, D. Buntinas, S. Kini, D. K. Panda, and P. Wyckoff. Microbenchmark performance comparison of high-speed cluster interconnects. IEEE Micro, 24:42--51, January 2004.

Digital Library

[12]

J. Reid. The New Features of Fortran 2003. SIGPLAN Fortran Forum, 26:10--33, April 2007.

Digital Library

[13]

J. Stanton. On the vibronic level structure in the no radical. i. the ground electronic state. The Journal of Chemical Physics, Jan 2007.

[14]

J. Stanton. On the vibronic level structure in the no3 radical: Ii. adiabatic calculation of the infrared.... Molecular Physics, Jan 2009.

[15]

TACC. Longhorn User Guide. http://www.tacc.utexas.edu/user-services/user-guides/longhorn-user-guide.

[16]

G. Varadhan and D. Manocha. Out-of-core rendering of massive geometric environments. In Visualization, 2002. VIS 2002. IEEE, pages 69--76, nov. 2002.

Digital Library

[17]

G. V. Vaughan, B. Elliston, T. Tromey, and I. L. Taylor. GNU Autoconf, Automake, and Libtool. Pearson Education, Oct. 2000.

Digital Library

Cited By

Gallardo EVienne JFialho LTeller PBrowne J(2018)Employing MPI_T in MPI Advisor to optimize application performanceInternational Journal of High Performance Computing Applications10.1177/109434201668400532:6(882-896)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1177/1094342016684005
Gallardo EVienne JFialho LTeller PBrowne JDongarra JDenis AGoglin BJeannot EMercier G(2015)MPI AdvisorProceedings of the 22nd European MPI Users' Group Meeting10.1145/2802658.2802667(1-10)Online publication date: 21-Sep-2015
https://dl.acm.org/doi/10.1145/2802658.2802667
Gómez-Iglesias APekurovsky DHamidouche KZhang JVienne JPeterson G(2015)Porting scientific libraries to PGAS in XSEDE resourcesProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792785(1-7)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792785
Show More Cited By

Recommendations

New capabilities in qoscosgrid middleware for advanced job management, advance reservation and co-allocation of computing resources --- quantum chemistry application use case
Building a National Distributed e-Infrastructure - PL-Grid

In this chapter we present the new capabilities of QosCosGrid (QCG) middleware for advanced job and resource management in the grid environment. By connecting many computing clusters together, QosCosGrid offers easy-to-use mapping, execution and ...
Exploiting Distributed-Memory and Shared-Memory Parallelism on Clusters of SMPs with Data Parallel Programs

Clusters of SMPs are hybrid-parallel architectures that combine the main concepts of distributed-memory and shared-memory parallel machines. Although SMP clusters are widely used in the high performance computing community, there exists no single ...
High-Performance Design of YARN MapReduce on Modern HPC Clusters with Lustre and RDMA
IPDPS '15: Proceedings of the 2015 IEEE International Parallel and Distributed Processing Symposium

The viability and benefits of running MapReduce over modern High Performance Computing (HPC) clusters, with high performance interconnects and parallel file systems, have attracted much attention in recent times due to its uniqueness of solving data ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

XSEDE '12: Proceedings of the 1st Conference of the Extreme Science and Engineering Discovery Environment: Bridging from the eXtreme to the campus and beyond

July 2012

423 pages

ISBN:9781450316026

DOI:10.1145/2335755

General Chair:
Craig Stewart
Indiana University

Copyright © 2012 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 16 July 2012

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

XSEDE12

XSEDE12: 2012 eXtreme Science and Engineering Discovery Environment 2012

July 16 - 20, 2012

Illinois, Chicago, USA

Acceptance Rates

Overall Acceptance Rate 129 of 190 submissions, 68%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

4
Total Citations
View Citations
154
Total Downloads

Downloads (Last 12 months)1
Downloads (Last 6 weeks)0

Reflects downloads up to 16 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

Gallardo EVienne JFialho LTeller PBrowne J(2018)Employing MPI_T in MPI Advisor to optimize application performanceInternational Journal of High Performance Computing Applications10.1177/109434201668400532:6(882-896)Online publication date: 1-Nov-2018
https://dl.acm.org/doi/10.1177/1094342016684005
Gallardo EVienne JFialho LTeller PBrowne JDongarra JDenis AGoglin BJeannot EMercier G(2015)MPI AdvisorProceedings of the 22nd European MPI Users' Group Meeting10.1145/2802658.2802667(1-10)Online publication date: 21-Sep-2015
https://dl.acm.org/doi/10.1145/2802658.2802667
Gómez-Iglesias APekurovsky DHamidouche KZhang JVienne JPeterson G(2015)Porting scientific libraries to PGAS in XSEDE resourcesProceedings of the 2015 XSEDE Conference: Scientific Advancements Enabled by Enhanced Cyberinfrastructure10.1145/2792745.2792785(1-7)Online publication date: 26-Jul-2015
https://dl.acm.org/doi/10.1145/2792745.2792785
Gómez-Iglesias AVienne JHamidouche KSimmons CBarth WPanda D(2015)Scalable Out-of-core OpenSHMEM Library for HPCRevised Selected Papers of the Second Workshop on OpenSHMEM and Related Technologies. Experiences, Implementations, and Technologies - Volume 939710.1007/978-3-319-26428-8_9(138-153)Online publication date: 4-Aug-2015
https://dl.acm.org/doi/10.1007/978-3-319-26428-8_9

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents