research-article

Public Access

An efficient MPI/openMP parallelization of the Hartree-Fock method for the second generation of Intel^® Xeon Phi^™ processor

Authors:

Vladimir Mironov,

Kristopher Keipert,

Michael D'mello,

Alexander Moskovsky,

Mark S. GordonAuthors Info & Claims

SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 39, Pages 1 - 12

https://doi.org/10.1145/3126908.3126956

Published: 12 November 2017 Publication History

Abstract

Modern OpenMP threading techniques are used to convert the MPI-only Hartree-Fock code in the GAMESS program to a hybrid MPI/OpenMP algorithm. Two separate implementations that differ by the sharing or replication of key data structures among threads are considered, density and Fock matrices. All implementations are benchmarked on a super-computer of 3,000 Intel® Xeon Phi^™ processors. With 64 cores per processor, scaling numbers are reported on up to 192,000 cores. The hybrid MPI/OpenMP implementation reduces the memory footprint by approximately 200 times compared to the legacy code. The MPI/OpenMP code was shown to run up to six times faster than the original for a range of molecular system sizes.

References

[1]

2017. Argonne National Laboratory Leadership Computing Facility. (2017). http://www.alcf.anl.gov/

[2]

2017. The General Atomic and Molecular Electronic Structure System (GAMESS) is a general ab initio quantum chemistry package. (2017). http://www.msg.ameslab.gov/gamess/index.html

[3]

2017. U.S. Department of Energy Office of Science Innovative and Novel Computational Impact on Theory and Experiment (INCITE) program. (2017). http://www.doeleadershipcomputing.org/

[4]

Yuri Alexeev, Ricky A Kendall, and Mark S Gordon. 2002. The distributed data SCF. Computer Physics Communications 143, 1 (2002), 69--82.

[5]

Yuri Alexeev, Ashutosh Mahajan, Sven Leyffer, Graham Fletcher, and Dmitri G Fedorov. 2012. Heuristic static load-balancing algorithm applied to the fragment molecular orbital method. In International Conference for High Performance Computing, Networking, Storage and Analysis (SC'12). IEEE, 1--13.

Digital Library

[6]

Yuri Alexeev, Michael W Schmidt, Theresa L Windus, and Mark S Gordon. 2007. A parallel distributed data CPHF algorithm for analytic Hessians. Journal of Computational Chemistry 28, 10 (2007), 1685--1694.

[7]

Edoardo Apra, Michael Klemm, and Karol Kowalski. 2014. Efficient implementation of many-body quantum chemical methods on the Intel® Xeon Phi co-processor. In 2014 International Conference for High Performance Computing, Networking, Storage and Analysis (SC'14). IEEE Press, 674--684.

Digital Library

[8]

Andrey Asadchev and Mark S Gordon. 2012. New multithreaded hybrid CPU/GPU approach to Hartree-Fock. Journal of Chemical Theory and Computation 8, 11 (2012), 4166--4176.

[9]

Edmond Chow, Xing Liu, Sanchit Misra, Marat Dukhan, Mikhail Smelyanskiy, Jeff R Hammond, Yunfei Du, Xiang-Ke Liao, and Pradeep Dubey. 2015. Scaling up Hartree-Fock calculations on Tianhe-2. International Journal of High Performance Computing Applications (2015), 85--102.

Digital Library

[10]

Edmond Chow, Xing Liu, Mikhail Smelyanskiy, and Jeff R Hammond. 2015. Parallel scalability of Hartree-Fock calculations. The Journal of Chemical Physics 142, 10 (2015), 104103.

[11]

Dmitri G Fedorov and Kazuo Kitaura. 2007. Extending the power of quantum chemistry to large systems with the fragment molecular orbital method. The Journal of Physical Chemistry A 111, 30 (2007), 6904--6914.

[12]

D G Fedorov and K Kitaura (Eds.). 2009. The Fragment Molecular Orbital Method: Practical Applications to Large Molecular Systems. CRC Press, Boca Raton, FL, USA.

[13]

Graham D Fletcher, Michael W Schmidt, Brett M Bode, and Mark S Gordon. 2000. The distributed data interface in GAMESS. Computer Physics Communications 128, 1 (2000), 190--200.

[14]

Ian T Foster, Jeffrey L Tilson, Albert F Wagner, Ron L Shepard, Robert J Harrison, Rick A Kendall, and Rik J Littlefield. 1996. Toward high-performance computational chemistry: I. Scalable Fock matrix construction algorithms. Journal of Computational Chemistry 17, 1 (1996), 109--123.

[15]

Mark S Gordon and Michael W. Schmidt. 2005. Advances in electronic structure theory: GAMESS a decade later. In Theory and Applications of Computational Chemistry: the first forty years, CE Dykstra, G. Frenking, KS Kim, and GE Scuseria (Eds.). Elsevier, Amsterdam, 1167--1189.

[16]

Robert J Harrison, Martyn F Guest, Rick A Kendall, David E Bernholdt, Adrian T Wong, Mark Stave, James L Anchell, Anthony C Hess, Rik J Littlefield, George L Fann, and others. 1996. Toward high-performance computational chemistry: II A Scalable Self Consitent Field Program. Journal of Computational Chemistry 17, 1 (1996), 124--132.

[17]

Kazuya Ishimura, Kei Kuramoto, Yasuhiro Ikuta, and Shi-aki Hyodo. 2010. MPI/OpenMP hybrid parallel algorithm for hartree- fock calculations. Journal of Chemical Theory and Computation 6, 4 (2010), 1075--1080.

[18]

Curtis L. Janssen and Ida M.B. Nielsen. 2008. Parallel computing in quantum chemistry. CRC Press, Boca Raton, FL, USA.

[19]

Shigeki Kawai, Andrea Benassi, Enrico Gnecco, Hajo Söde, Rémy Pawlak, Xinliang Feng, Klaus Müllen, Daniele Passerone, Carlo A Pignedoli, Pascal Ruffieux, and others. 2016. Superlubricity of graphene nanoribbons on gold surfaces. Science 351, 6276 (2016), 957--961.

[20]

Xing Liu, Anup Patel, and Edmond Chow. 2014. A new scalable parallel algorithm for Fock matrix construction. In Parallel and Distributed Processing Symposium, 2014 IEEE 28th International. IEEE, 902--914.

Digital Library

[21]

Vladimir Mironov, Maria Khrenova, and Alexander Moskovsky. 2015. On Quantum Chemistry Code Adaptation for RSC PetaStream Architecture. In High Performance Computing: 30th International Conference, ISC High Performance 2015, Frankfurt, Germany, July 12-16, 2015, Proceedings. Springer, 113--121.

[22]

David Ozog, Amir Kamil, Yili Zheng, Paul Hargrove, Jeff R Hammond, Allen Malony, Wibe de Jong, and Kathy Yelick. 2016. A Hartree-Fock application using UPC++ and the new DArray library. In Parallel and Distributed Processing Symposium, 2016 IEEE International. IEEE, 453--462.

[23]

Spencer Pruit. 2016. private communication. (2016).

[24]

Michael W Schmidt, Kim K Baldridge, Jerry A Boatz, Steven T Elbert, Mark S Gordon, Jan H Jensen, Shiro Koseki, Nikita Matsunaga, Kiet A Nguyen, Shujun Su, and others. 1993. General Atomic and Molecular Electronic Structure System. Journal of Computational Chemistry 14, 11 (1993), 1347--1363.

Digital Library

[25]

Ivan S Ufimtsev and Todd J Martinez. 2008. Quantum chemistry on graphical processing units. 1. Strategies for two-electron integral evaluation. Journal of Chemical Theory and Computation 4, 2 (2008), 222--231.

[26]

Ivan S Ufimtsev and Todd J Martinez. 2009. Quantum chemistry on graphical processing units. 2. Direct self-consistent-field implementation. Journal of Chemical Theory and Computation 5, 4 (2009), 1004--1015.

[27]

Hiroaki Umeda, Yuichi Inadomi, Toshio Watanabe, Toru Yagi, Takayoshi Ishimoto, Tsutomu Ikegami, Hiroto Tadano, Tetsuya Sakurai, and Umpei Nagashima. 2010. Parallel Fock matrix construction with distributed shared memory model for the FMO-MO method. Journal of Computational Chemistry 31, 13 (2010), 2381--2388.

[28]

Marat Valiev, Eric J Bylaska, Niranjan Govind, Karol Kowalski, Tjerk P Straatsma, Hubertus JJ Van Dam, Dunyou Wang, Jarek Nieplocha, Edoardo Apra, Theresa L Windus, and others. 2010. NWChem: A comprehensive and scalable open-source solution for large scale molecular simulations. Computer Physics Communications 181, 9 (2010), 1477--1489.

[29]

Karl A Wilkinson, Paul Sherwood, Martyn F Guest, and Kevin J Naidoo. 2011. Acceleration of the GAMESS-UK electronic structure package on graphical processing units. Journal of Computational Chemistry 32, 10 (2011), 2313--2318.

Cited By

Ito YTsuji SFujii HSuzuki KYokogawa NNakano KKasagi A(2024)Introduction to Computational Quantum Chemistry for Computer Scientists2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW63119.2024.00066(273-282)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPSW63119.2024.00066
Yan XLiu CYang MFeng WZhong M(2023)An Improved Parameter Estimation Method for High-Efficiency Multi-GNSS-Integrated Orbit DeterminationRemote Sensing10.3390/rs1510263515:10(2635)Online publication date: 18-May-2023
https://doi.org/10.3390/rs15102635
Alkan MPham BHammond JGordon M(2023)Enabling Fortran Standard Parallelism in GAMESS for Accelerated Quantum Chemistry CalculationsJournal of Chemical Theory and Computation10.1021/acs.jctc.3c0038019:13(3798-3805)Online publication date: 21-Jun-2023
https://doi.org/10.1021/acs.jctc.3c00380
Show More Cited By

Index Terms

An efficient MPI/openMP parallelization of the Hartree-Fock method for the second generation of Intel^® Xeon Phi^™ processor
1. Computing methodologies
  1. Modeling and simulation
    1. Simulation types and techniques
      1. Massively parallel and high-performance simulations
      2. Quantum mechanic simulation
2. Theory of computation
  1. Design and analysis of algorithms
    1. Parallel algorithms
      1. Massively parallel algorithms

Recommendations

An efficient MPI/OpenMP parallelization of the Hartree-Fock-Roothaan method for the first generation of Intel® Xeon Phiý processor architecture

The Hartree-Fock method in the General Atomic and Molecular Structure System GAMESS quantum chemistry package represents one of the most irregular algorithms in computation today. Major steps in the calculation are the irregular computation of electron ...
Explicit Fourth-Order Runge---Kutta Method on Intel Xeon Phi Coprocessor

This paper concerns an Intel Xeon Phi implementation of the explicit fourth-order Runge---Kutta method (RK4) for very sparse matrices with very short rows. Such matrices arise during Markovian modeling of computer and telecommunication networks. In this ...
Direct MPI Library for Intel Xeon Phi Co-Processors
IPDPSW '13: Proceedings of the 2013 IEEE 27th International Symposium on Parallel and Distributed Processing Workshops and PhD Forum

DCFA-MPI is an MPI library implementation for Intel Xeon Phi co-processor clusters, where a compute node consists of an Intel Xeon Phi co-processor card connected to the host via PCI Express with InfiniBand. DCFA-MPI enables direct data transfer between ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2017

801 pages

ISBN:9781450351140

DOI:10.1145/3126908

General Chair:
Bernd Mohr
Jülich Supercomputing Center, Jülich, Germany
,
Program Chair:
Padma Raghavan
Vanderbilt University, Nashville, TN

Copyright © 2017 ACM.

© 2017 Association for Computing Machinery. ACM acknowledges that this contribution was authored or co-authored by an employee, contractor or affiliate of the United States government. As such, the United States Government retains a nonexclusive, royalty-free right to publish or reproduce this article, or to allow others to do so, for Government purposes only.

Sponsors

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

In-Cooperation

IEEE CS

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 12 November 2017

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Funding Sources

Conference

SC '17

Sponsor:

SIGHPC

SC '17: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2017

Colorado, Denver

Acceptance Rates

SC '17 Paper Acceptance Rate 61 of 327 submissions, 19%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Upcoming Conference

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

12
Total Citations
View Citations
632
Total Downloads

Downloads (Last 12 months)138
Downloads (Last 6 weeks)16

Reflects downloads up to 28 Dec 2024

Other Metrics

View Author Metrics

Citations

Cited By

Ito YTsuji SFujii HSuzuki KYokogawa NNakano KKasagi A(2024)Introduction to Computational Quantum Chemistry for Computer Scientists2024 IEEE International Parallel and Distributed Processing Symposium Workshops (IPDPSW)10.1109/IPDPSW63119.2024.00066(273-282)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPSW63119.2024.00066
Yan XLiu CYang MFeng WZhong M(2023)An Improved Parameter Estimation Method for High-Efficiency Multi-GNSS-Integrated Orbit DeterminationRemote Sensing10.3390/rs1510263515:10(2635)Online publication date: 18-May-2023
https://doi.org/10.3390/rs15102635
Alkan MPham BHammond JGordon M(2023)Enabling Fortran Standard Parallelism in GAMESS for Accelerated Quantum Chemistry CalculationsJournal of Chemical Theory and Computation10.1021/acs.jctc.3c0038019:13(3798-3805)Online publication date: 21-Jun-2023
https://doi.org/10.1021/acs.jctc.3c00380
Lykov DSchutski RGalda AVinokur VAlexeev Y(2022)Tensor Network Quantum Simulator With Step-Dependent Parallelization2022 IEEE International Conference on Quantum Computing and Engineering (QCE)10.1109/QCE53715.2022.00081(582-593)Online publication date: Oct-2022
https://doi.org/10.1109/QCE53715.2022.00081
Seidl CBarca G(2022)Q-Next: A Fast, Parallel, and Diagonalization-Free Alternative to Direct Inversion of the Iterative SubspaceJournal of Chemical Theory and Computation10.1021/acs.jctc.2c0007318:7(4164-4176)Online publication date: 24-Jun-2022
https://doi.org/10.1021/acs.jctc.2c00073
Chen XGe MHugentobler USchuh H(2022)A new parallel algorithm for improving the computational efficiency of multi-GNSS precise orbit determinationGPS Solutions10.1007/s10291-022-01266-826:3Online publication date: 27-May-2022
https://doi.org/10.1007/s10291-022-01266-8
Pham BDatta DGordon M(2021)PDG: A Composite Method Based on the Resolution of the IdentityThe Journal of Physical Chemistry A10.1021/acs.jpca.1c06186125:42(9421-9429)Online publication date: 18-Oct-2021
https://doi.org/10.1021/acs.jpca.1c06186
Chapman BPham BYang CDaley CBertoni CKulkarni DOryspayev DD’Azevedo EDoerfert JZhou KRavikumar KGordon MDel Ben MLin MAlkan MKruse MHernandez OYeung PLin PXu PPophale SSattasathuchana TKale VHuhn WHe Y(2021)Outcomes of OpenMP Hackathon: OpenMP Application Experiences with the Offloading Model (Part II)OpenMP: Enabling Massive Node-Level Parallelism10.1007/978-3-030-85262-7_6(81-95)Online publication date: 8-Sep-2021
https://doi.org/10.1007/978-3-030-85262-7_6
Li MHawrylak PHale JFurlani T(2019)Combining OpenCL and MPI to Support Heterogeneous Computing on a ClusterPractice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)10.1145/3332186.3333059(1-6)Online publication date: 28-Jul-2019
https://dl.acm.org/doi/10.1145/3332186.3333059
Huang HChow E(2018)Accelerating quantum chemistry with vectorized and batched integralsProceedings of the International Conference for High Performance Computing, Networking, Storage, and Analysis10.5555/3291656.3291711(1-14)Online publication date: 11-Nov-2018
https://dl.acm.org/doi/10.5555/3291656.3291711
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents