research-article

Extending stability beyond CPU millennium: a micron-scale atomistic simulation of Kelvin-Helmholtz instability

Authors:

D. F. Richards,

K. J. Caspersen,

F. H. StreitzAuthors Info & Claims

SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing

Article No.: 58, Pages 1 - 11

https://doi.org/10.1145/1362622.1362700

Published: 10 November 2007 Publication History

Abstract

We report the computational advances that have enabled the first micron-scale simulation of a Kelvin-Helmholtz (KH) instability using molecular dynamics (MD). The advances are in three key areas for massively parallel computation such as on BlueGene/L (BG/L): fault tolerance, application kernel optimization, and highly efficient parallel I/O. In particular, we have developed novel capabilities for handling hardware parity errors and improving the speed of interatomic force calculations, while achieving near optimal I/O speeds on BG/L, allowing us to achieve excellent scalability and improve overall application performance. As a result we have successfully conducted a 2-billion atom KH simulation amounting to 2.8 CPU-millennia of run time, including a single, continuous simulation run in excess of 1.5 CPU-millennia. We have also conducted 9-billion and 62.5-billion atom KH simulations. The current optimized ddcMD code is benchmarked at 115.1 TFlop/s in our scaling study and 103.9 TFlop/s in a sustained science run, with additional improvements ongoing. These improvements enabled us to run the first MD simulations of micron-scale systems developing the KH instability.

References

[1]

M. P. Allen and D. J. Tildesley. Computer Simulation of Liquids. Clarendon Press, Oxford, 1987.

Digital Library

[2]

S. Chandrasekhar. Hydrodynamic and Hydromagnetic Stability. Oxford Univ. Press, Oxford, 1961.

[3]

M. P. Ciamarra, A. Coniglio, and M. Micodemi. Shear instabilities in granular mixtures. Phys. Rev. Lett., 94:188001, 2005.

[4]

P. Crozier, F. Reid, and C. Vaughn. Lammps benchmarks, http://lammps.sandia.gov/bench.html.

[5]

M. S. Daw and M. I. Baskes. Embedded-atom method: Derivation and application to impurities, surfaces, and other defects in metals. Phys. Rev. B, 29(12):6443--6453, 1984.

[6]

P. G. Drazin and W. H. Reid. Hydrodynamic Stability. Cambridge Univ. Press, Cambridge, 1981.

[7]

E. N. M. Elnozahy, L. Alvisi, Y.-M. Wang, and D. B. Johnson. A survey of rollback-recovery protocols in message-passing systems. ACM Comput. Surv., 34(3):375--408, 2002.

Digital Library

[8]

C. Engelmann and A. Geist. Super-scalable algorithms for computing on 100,000 processors. In Computational Science - ICCS 2005, PT 1, Proceedings, volume 3514 of Lecture Notes in Computer Science, pages 313--321, 2005.

Digital Library

[9]

O. B. Fringer and R. L. Street. The dynamics of breaking progressive interfacial waves. J. Fluid Mech., 494:319--353, 2003.

[10]

D. C. Fritts, C. Bizon, J. A. Werne, and C. K. Meyer. Layering accompanying turbulence generation due to shear instability and gravity-wave breaking. J. Geophys. Res., 108:20:1--13, 2003.

[11]

D. C. Fritts, T. L. Palmer, O. Andreassen, and I. Lie. Evolution and breakdown of kelvin-helmholtz billows in stratified compressible flows:1, comparison of two-and three-dimensional flows. J. Atmos. Sci., 53:3173--3191, 1996.

[12]

T. C. Germann, K. Kadau, and P. S. Lomdahl. 25 tflop/s multibillion-atom molecular dynamics simulations and visualization/analysis on bluegene/1. Proc. Supercomputing 2005, Seattle, 2005, on CD-ROM, http://sc05.supercomputing.org/schedule/pdf/papl22.pdf.

[13]

M. Gomaa, C. Scarbrough, T. N. Vijaykumar, and I. Pomeranz. Transient-fault recovery for chip multiprocessors. In ISCA '03: Proceedings of the 30th annual international symposium on Computer architecture, pages 98--109, New York, NY, USA, 2003. ACM Press.

Digital Library

[14]

Grape newletter vol.8 (dec 16 2006), http://grape.mtk.nao.ac.jp/grape/newsletter/061216.html.

[15]

J. F. Hawley and N. J. Zabusky. Vortex paradigm for shock-accelerated density-stratified interfaces. Phys. Rev. Lett., 63(12): 1241--1244, 1989.

[16]

J. K. Holt, H. G. Park, Y. Wang, M. Stadermann, A. B. Artyukhin, C. P. Grigoropoulos, A. Noy, and O. Bakajin. Fast Mass Transport Through Sub-2-Nanometer Carbon Nanotubes. Science, 312(5776):1034--1037, 2006.

[17]

R. A. Johnson. Alloy models with the embedded-atom method. Phys. Rev. B, 39:12554, 1989.

[18]

R. A. Johnson and D. J. Oh. Analytic embedded atom method model for bcc metals. J. Mater. Res., 4:1195, 1989.

[19]

V. Junk, F. Heitsch, and T. Naab. The kelvin-helmholtz instability in smoothed particle hydrodynamics. Proc. Int. Astro. Union, 2:210--210, 2007.

[20]

K. Kadau, T. C. Germann, N. G. Hadjiconstantinou, P. S. Lomdahl, G. Dimonte, B. L. Holian, and B. J. Alder. Nanohydrodynamics simulations: An atomistic view of the rayleigh-taylor instability. Proc. Natl. Acad. Sci. USA, 101:5851--5855, 2004.

[21]

K. Kadau, T. C. Germann, and P. S. Lamdahl. Molecular dynamics comes of age: 320 billion atom simulation on bluegene/1. Int. J. Mod. Phys. C, 17:1755, 2006.

[22]

H. Lamb. Hydrodynamics. Cambridge Univ. Press, Cambridge, 6th edition, 1932.

[23]

D. R. Mason, R. E. Rudd, and A. P. Sutton. Stochastic kinetic monte carlo algorithms for long-range hamiltonians. Computer Physics Comm., 160:140--157, 2004.

[24]

J. J. Monaghan. Flaws in the modern laplacian theory. Earth, Moon and Planets, 71:73--84, 1995.

[25]

J. A. Moriarty. Analytic representation of multi-ion interatomic potentials in transition metals. Phys. Rev. B, 42:1609--1628, 1990.

[26]

J. A. Moriarty. Angular forces and melting in bcc transition metals: A case study of molybdenum. Phys. Rev. B, 49:12431--12445, 1994.

[27]

T. Narumi, Y. Ohno, N. Okimoto, T. Koishi, A. Suenaga, N. Futatsugi, R. Yanai, R. Himeno, S. Fujikawa, M. Taiji, and M. Ikei. A 55 tflops simulation of amyloid-forming peptides from yeast prion sup35 with the special-purpose computer system mdgrape-3. Proc. Supercomputing 2006, Tampa, 2006, on CD-ROM, http://sc06.supercomputing.org/schedule/pdf/gb106.pdf.

Digital Library

[28]

T. L. Palmer, D. C. Fritts, and O. Andreassen. Evolution and breakdown of kelvin-helmholtz billows in stratified compressible flows:2. instability structure, evolution, and energetics. J. Atmos. Sci., 53:3192--3212, 1996.

[29]

D. C. Rapaport. The Art of Molecular Dynamics Simulation. Cambridge Univ. Press, Cambridge, 1995.

Digital Library

[30]

http://www.sandia.gov/asc/redstorm.html.

[31]

G. A. Reis, J. Chang, N. Vachharajani, R. Rangan, and D. I. August. Swift: Software implemented fault tolerance. In CGO '05: Proceedings of the international symposium on Code generation and optimization, pages 243--254, Washington, DC, USA, 2005. IEEE Computer Society.

Digital Library

[32]

G. A. Reis, J. Chang, N. Vachharajani, R. Rangan, D. I. August, and S. S. Mukherjee. Design and evaluation of hybrid fault-detection systems. SIGARCH Comput. Archit. News, 33(2):148--159, 2005.

Digital Library

[33]

R. E. Rudd and J. Q. Broughton. Coupling of length scales in solid state systems. Phys. Stat. Sol., 217:251--291, 2000.

[34]

SETI@HOME project: http://setiathome.ssl.berkeley.edu.

[35]

F. H. Streitz, J. N. Glosli, and M. V. Patel. Beyond finite-size scaling in solidification simulations. Phys. Rev. Lett., 96:225701, 2006.

[36]

F. H. Streitz, J. N. Glosli, and M. V. Patel. Simulating solidification in metals at high pressure: The drive to petascale computing. J. Phys.: Conf. Ser., 46:254--267, 2006.

[37]

R. I. Sykes and W. S. Lewellen. A numerical study of breaking of kelvin-helmholtz billows using a reynolds-stress turbulence closure model. J. Atmos. Sci., 39:1506--1520, 1982.

[38]

TOP500 Supercomputer Sites: http://www.top500.org/.

[39]

J.-Y. Yang and J.-W. Chang. Rarefied flow instability simulation using model boltzmann equations. Proc. Fluid Dynamics Conf. 28th, Snowmass Village, CO, June 29-July 2, 1997.

[40]

Y. Yeh. Triple-triple redundant 777 primary flight computer. In Proceedings of the 1996 IEEE Aerospace Applications Conference, volume 1, pages 293--307, 1996.

[41]

R. Zhang, X. He, G. Doolen, and S. Chen. Surface tension effect on two-dimensional two-phase kelvin-helmholtz instabilities. Adv. Water Res., 24:461--478, 2001.

Cited By

Pachalieva APavel RSantos JDiaw ALubbers NMehana MHaack JViswanathan HLivescu DGermann TJunghans C(2022)GLUE Code: A framework handling communication and interfaces between scalesJournal of Open Source Software10.21105/joss.048227:80(4822)Online publication date: Dec-2022
https://doi.org/10.21105/joss.04822
Bhatia HDi Natale FMoon JZhang XChavez JAydin FStanley COppelstrup TNeale CSchumacher SAhn DHerbein SCarpenter TGnanakaran SBremer PGlosli JLightstone FIngólfsson Hde Supinski BHall MGamblin T(2021)Generalizable coordination of large multiscale workflowsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476210(1-16)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3458817.3476210
Bhatia HCarpenter TIngólfsson HDharuman GKarande PLiu SOppelstrup TNeale CLightstone FVan Essen BGlosli JBremer P(2021)Machine-learning-based dynamic-importance sampling for adaptive multiscale simulationsNature Machine Intelligence10.1038/s42256-021-00327-w3:5(401-409)Online publication date: 22-Apr-2021
https://doi.org/10.1038/s42256-021-00327-w
Show More Cited By

Recommendations

Beyond B-Cell Epitopes: Curating Positive Data on Antipeptide Paratope Binding to Support Development of Computational Tools for Vaccine Design and Other Translational Applications
BCB '20: Proceedings of the 11th ACM International Conference on Bioinformatics, Computational Biology and Health Informatics

B-cell epitope prediction was first developed to help design peptide-based vaccines for protective antibody-mediated immunity exemplified by neutralization of biological activity (e.g., pathogen infectivity). Requisite computational tools are ...
Cellular automata beyond 100k cores: MPI vs Fortran coarrays
EuroMPI '18: Proceedings of the 25th European MPI Users' Group Meeting

Fortran coarrays are an attractive alternative to MPI due to a familiar Fortran syntax, single sided communications and implementation in the compiler. Scaling of coarrays is compared in this work to MPI, using cellular automata (CA) 3D Ising ...
Experience of parallelizing cryo-EM 3D reconstruction on a CPU-GPU heterogeneous system
HPDC '11: Proceedings of the 20th international symposium on High performance distributed computing

Heterogeneous architecture is becoming an important way to build a massive parallel computer system, i.e. the CPU-GPU heterogeneous systems ranked in Top500 list. However, it is a challenge to efficiently utilize massive parallelism of both applications ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '07: Proceedings of the 2007 ACM/IEEE conference on Supercomputing

November 2007

723 pages

ISBN:9781595937643

DOI:10.1145/1362622

General Chair:
Becky Verastegui
Oak Ridge National Laboratory

Copyright © 2007 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Sponsors

SIGARCH: ACM Special Interest Group on Computer Architecture
IEEE-CS: Computer Society

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 10 November 2007

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Qualifiers

Research-article

Funding Sources

U.S. Department of Energy

Conference

SC '07

Sponsor:

SIGARCH
IEEE-CS

SC '07: International Conference for High Performance Computing, Networking, Storage and Analysis

November 10 - 16, 2007

Nevada, Reno

Acceptance Rates

SC '07 Paper Acceptance Rate 54 of 268 submissions, 20%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

75
Total Citations
View Citations
584
Total Downloads

Downloads (Last 12 months)29
Downloads (Last 6 weeks)0

Reflects downloads up to 12 Sep 2024

Other Metrics

View Author Metrics

Citations

Cited By

Pachalieva APavel RSantos JDiaw ALubbers NMehana MHaack JViswanathan HLivescu DGermann TJunghans C(2022)GLUE Code: A framework handling communication and interfaces between scalesJournal of Open Source Software10.21105/joss.048227:80(4822)Online publication date: Dec-2022
https://doi.org/10.21105/joss.04822
Bhatia HDi Natale FMoon JZhang XChavez JAydin FStanley COppelstrup TNeale CSchumacher SAhn DHerbein SCarpenter TGnanakaran SBremer PGlosli JLightstone FIngólfsson Hde Supinski BHall MGamblin T(2021)Generalizable coordination of large multiscale workflowsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476210(1-16)Online publication date: 14-Nov-2021
https://dl.acm.org/doi/10.1145/3458817.3476210
Bhatia HCarpenter TIngólfsson HDharuman GKarande PLiu SOppelstrup TNeale CLightstone FVan Essen BGlosli JBremer P(2021)Machine-learning-based dynamic-importance sampling for adaptive multiscale simulationsNature Machine Intelligence10.1038/s42256-021-00327-w3:5(401-409)Online publication date: 22-Apr-2021
https://doi.org/10.1038/s42256-021-00327-w
Ji HMascagni MLi Y(2020)Gaussian variant of Freivalds’ algorithm for efficient and reliable matrix product verificationMonte Carlo Methods and Applications10.1515/mcma-2020-207626:4(273-284)Online publication date: 8-Oct-2020
https://doi.org/10.1515/mcma-2020-2076
Parasyris KKeller KBautista-Gomez LUnsal O(2020)Checkpoint Restart Support for Heterogeneous HPC Applications2020 20th IEEE/ACM International Symposium on Cluster, Cloud and Internet Computing (CCGRID)10.1109/CCGrid49817.2020.00-69(242-251)Online publication date: May-2020
https://doi.org/10.1109/CCGrid49817.2020.00-69
Ahn DBass NChu AGarlick JGrondona MHerbein SIngólfsson HKoning JPatki TScogland TSpringmeyer BTaufer M(2020)Flux: Overcoming scheduling challenges for exascale workflowsFuture Generation Computer Systems10.1016/j.future.2020.04.006Online publication date: Apr-2020
https://doi.org/10.1016/j.future.2020.04.006
Di Natale FBhatia HCarpenter TNeale CKokkila-Schumacher SOppelstrup TStanton LZhang XSundram SScogland TDharuman GSurh MYang YMisale CSchneidenbach LCosta CKim CD'Amora BGnanakaran SNissley DStreitz FLightstone FBremer PGlosli JIngólfsson HTaufer MBalaji PPeña A(2019)A massively parallel infrastructure for adaptive multiscale simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3295500.3356197(1-16)Online publication date: 17-Nov-2019
https://dl.acm.org/doi/10.1145/3295500.3356197
Patki TFrye ZBhatia HDi Natale FGlosli JIngolfsson HRountree B(2019)Comparing GPU Power and Frequency Capping: A Case Study with the MuMMI Workflow2019 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS)10.1109/WORKS49585.2019.00009(31-39)Online publication date: Nov-2019
https://doi.org/10.1109/WORKS49585.2019.00009
Lu Y(2019)Paving the way for China exascale computingCCF Transactions on High Performance Computing10.1007/s42514-019-00010-yOnline publication date: 9-Aug-2019
https://doi.org/10.1007/s42514-019-00010-y
Agarwal PNaughton TPark BBernholdt DHursey JGeist A(2019)Application health monitoring for extreme‐scale resiliency using cooperative fault managementConcurrency and Computation: Practice and Experience10.1002/cpe.544932:2Online publication date: 25-Jul-2019
https://doi.org/10.1002/cpe.5449
Show More Cited By

View Options

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Media

Figures

Other

Tables

View Table of Contents