research-article

Open access

69.7-PFlops Extreme Scale Earthquake Simulation with Crossing Multi-faults and Topography on Sunway

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 10, Pages 1 - 15

https://doi.org/10.1145/3581784.3613209

Published: 11 November 2023 Publication History

Abstract

A high-scalable and fully optimized earthquake model is presented based on the latest Sunway supercomputer. Contributions include: 1) the curvilinear grid finite-difference method (CGFDM) and flexible model applying perfectly matched layer (PML) and enabling more accurate and realistic terrain descriptions; 2) a hybrid and non-uniform domain decomposition scheme that efficiently maps the model across different levels of the computing system; and 3) sophisticated optimizations that largely alleviate or even eliminate bottlenecks in memory, communication, etc., obtaining a speedup of over 140×. Combining all innovations, the design fully exploits the hardware potential of all aspects and enables us to perform the largest CGFDM-based earthquake simulation ever reported (69.7 PFlops using over 39 million cores). Based on our design, the Turkey earthquakes (February 6, 2023), and the Ridgecrest earthquake (July 4, 2019), are successfully simulated with a maximum resolution of 12-m. Precise hazard evaluations for the hazardous reduction of earthquake-stricken areas are also conducted.

References

[1]

J Bielak. 1998. Ground motion modeling using 3D finite element methods. Proc. 2nd Intl. Sympo. Effects of Surface Geology on Seismic Motion, 1998 1 (1998), 121--133.

[2]

Jacobo Bielak, Omar Ghattas, and EJ Kim. 2005. Parallel octree-based finite element method for large-scale earthquake ground motion simulation. Computer Modeling in Engineering and Sciences 10, 2 (2005), 99.

[3]

Alexander Breuer, Alexander Heinecke, and Yifeng Cui. 2017. EDGE: Extreme Scale Fused Seismic Simulations with the Discontinuous Galerkin Method. In International Supercomputing Conference. Springer, 41--60.

[4]

Bingwei Chen, Haohuan Fu, Yanwen Wei, Conghui He, Wenqiang Zhang, Yuxuan Li, Wubin Wan, Wei Zhang, Lin Gan, Zhenguo Zhang, et al. 2018. Simulating the Wenchuan earthquake with accurate surface topography on Sunway TaihuLight. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 517--528.

Digital Library

[5]

Josep de la Puente, Martin Käser, Michael Dumbser, and Heiner Igel. 2007. An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes-IV. Anisotropy. Geophysical Journal International 169, 3 (2007), 1210--1228.

[6]

Michael Dumbser and Martin Käser. 2006. An arbitrary high-order discontinuous Galerkin method for elastic waves on unstructured meshes---II. The three-dimensional isotropic case. Geophysical Journal International 167, 1 (2006), 319--336.

[7]

Haohuan Fu, Conghui He, Bingwei Chen, Zekun Yin, Zhenguo Zhang, Wenqiang Zhang, Tingjian Zhang, Wei Xue, Weiguo Liu, Wanwang Yin, et al. 2017. 18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 2.

Digital Library

[8]

Haohuan Fu, Junfeng Liao, Jinzhe Yang, Lanning Wang, Zhenya Song, Xiaomeng Huang, Chao Yang, Wei Xue, Fangfang Liu, Fangli Qiao, et al. 2016. The Sunway TaihuLight supercomputer: system and applications. Science China Information Sciences 59, 7 (2016), 1--16.

[9]

Sezim Ezgi Güvercin, Hayrullah Karabulut, A Özgün Konca, Uğur Doğan, and Semih Ergintav. 2022. Active seismotectonics of the East Anatolian Fault. Geophysical Journal International 230, 1 (2022), 50--69.

[10]

Ufuk Hancılar, Karin Şeşetyan, Eser Çaktı, Erdal Şafak, Nesrin Yenihayat, Fatma S. Malcıoģlu, Kökcan Dönmez, Tuģce Tetik, Hakan Süleyman Gómez Capera, and Augusto Antonio. 2023. Kahramanmaraş-Gaziantep Türkiye M7. 7 Earthquake, 6 February 2023 (04: 17 GMT+ 03: 00): Strong Ground Motion and Building Damage Estimations Preliminary Report (v6). (2023).

[11]

Stephen H Hartzell, David L Carver, and Kenneth W King. 1994. Initial investigation of site and topographic effects at Robinwood Ridge, California. Bulletin of the Seismological Society of America 84, 5 (1994), 1336--1349.

[12]

Alexander Heinecke, Alexander Breuer, Sebastian Rettenberger, Michael Bader, Alice-Agnes Gabriel, Christian Pelties, Arndt Bode, William Barth, Xiang-Ke Liao, Karthikeyan Vaidyanathan, et al. 2014. Petascale high order dynamic rupture earthquake simulations on heterogeneous supercomputers. In High Performance Computing, Networking, Storage and Analysis, SC14: International Conference for. IEEE, 3--14.

[13]

Ray Hixon. 1997. On increasing the accuracy of MacCormack schemes for aeroacoustic applications. In 3rd AIAA/CEAS Aeroacoustics Conference. 1586.

[14]

Ray Hixon. 1998. Evaluation of a high-accuracy MacCormack-type scheme using benchmark problems. Journal of Computational Acoustics 6, 03 (1998), 291--305.

[15]

Tsuyoshi Ichimura, Kohei Fujita, Kentaro Koyama, Yuma Kikuchi, Ryota Kusakabe, Kazuo Minami, Hikaru Inoue, Seiya Nishizawa, Miwako Tsuji, Tatsuo Nishiki, et al. 2021. Fast scalable implicit solver with convergence of equation-based modeling and data-driven learning: earthquake city simulation on low-order unstructured finite element. In Proceedings of the Platform for Advanced Scientific Computing Conference. 1--12.

Digital Library

[16]

Tsuyoshi Ichimura, Kohei Fujita, Kentaro Koyama, Ryota Kusakabe, Yuma Kikuchi, Takane Hori, Muneo Hori, Lalith Maddegedara, Noriyuki Ohi, Tatsuo Nishiki, et al. 2022. 152K-computer-node parallel scalable implicit solver for dynamic nonlinear earthquake simulation. In International Conference on High Performance Computing in Asia-Pacific Region. 18--29.

Digital Library

[17]

Tsuyoshi Ichimura, Kohei Fujita, Ryota Kusakabe, Kentaro Koyama, Sota Murakami, Yuma Kikuchi, Takane Hori, Muneo Hori, Hikaru Inoue, Takafumi Nose, et al. 2022. Extreme scale earthquake simulation with uncertainty quantification. In 2022 SC22: International Conference for High Performance Computing, Networking, Storage and Analysis (SC). IEEE Computer Society, 37--47.

[18]

Tsuyoshi Ichimura, Kohei Fujita, Pher Errol Balde Quinay, Lalith Maddegedara, Muneo Hori, Seizo Tanaka, Yoshihisa Shizawa, Hiroshi Kobayashi, and Kazuo Minami. 2015. Implicit nonlinear wave simulation with 1.08 T DOF and 0.270 T unstructured finite elements to enhance comprehensive earthquake simulation. In Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. ACM, 4.

Digital Library

[19]

Tsuyoshi Ichimura, Kohei Fujita, Takuma Yamaguchi, Akira Naruse, Jack C Wells, Thomas C Schulthess, Tjerk P Straatsma, Christopher J Zimmer, Maxime Martinasso, Kengo Nakajima, et al. 2018. A fast scalable implicit solver for nonlinear time-evolution earthquake city problem on low-ordered unstructured finite elements with artificial intelligence and transprecision computing. In SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 627--637.

Digital Library

[20]

Saad Khan, Mark van der Meijde, Harald van der Werff, and Muhammad Shafique. 2020. The impact of topography on seismic amplification during the 2005 Kashmir earthquake. Natural Hazards and Earth System Sciences 20, 2 (2020), 399--411.

[21]

Dimitri Komatitsch and Jeroen Tromp. 1999. Introduction to the spectral element method for three-dimensional seismic wave propagation. Geophysical Journal International 139, 3 (1999), 806--822.

[22]

Dimitri Komatitsch, Seiji Tsuboi, Chen Ji, and Jeroen Tromp. 2003. A 14.6 billion degrees of freedom, 5 teraflops, 2.5 terabyte earthquake simulation on the Earth Simulator. In Supercomputing, 2003 ACM/IEEE Conference. IEEE, 4--4.

Digital Library

[23]

Dimitri Komatitsch and Jean-Pierre Vilotte. 1998. The spectral element method: an efficient tool to simulate the seismic response of 2D and 3D geological structures. Bulletin of the Seismological Society of America 88, 2 (1998), 368--392.

[24]

Diego Melgar, Tuncay Taymaz, Athanassios Ganas, Brendan Crowell, Taylan Öcalan, Metin Kahraman, Varvara Tsironi, Seda Yolsal-Çevikbilen, Sotiris Valkaniotis, Tahir Serkan Irmak, et al. 2023. Sub-and super-shear ruptures during the 2023 Mw 7.8 and Mw 7.6 earthquake doublet in SE Türkiye. Seismica 2, 3 (2023).

[25]

NG Reitman, RW Briggs, WD Barnhart, JA Thompson, CB DuRoss, AE Hatem, RD Gold, and JD Mejstrik. 2023. Preliminary fault rupture mapping of the 2023 M7. 8 and M7. 5 Türkiye Earthquakes.

[26]

Hannes Isaak Reuter, Andy Nelson, and Andrew Jarvis. 2007. An evaluation of void-filling interpolation methods for SRTM data. International Journal of Geographical Information Science 21, 9 (2007), 983--1008.

Digital Library

[27]

Max Rietmann, Peter Messmer, Tarje Nissen-Meyer, Daniel Peter, Piero Basini, Dimitri Komatitsch, Olaf Schenk, Jeroen Tromp, Lapo Boschi, and Domenico Giardini. 2012. Forward and adjoint simulations of seismic wave propagation on emerging large-scale GPU architectures. In SC'12: Proceedings of the International Conference on High Performance Computing, Networking, Storage and Analysis. IEEE, 1--11.

Digital Library

[28]

Zachary E Ross, Benjamín Idini, Zhe Jia, Oliver L Stephenson, Minyan Zhong, Xin Wang, Zhongwen Zhan, Mark Simons, Eric J Fielding, Sang-Ho Yun, et al. 2019. Hierarchical interlocked orthogonal faulting in the 2019 Ridgecrest earthquake sequence. Science 366, 6463 (2019), 346--351.

[29]

Daniel Roten, Yifeng Cui, Kim B Olsen, Steven M Day, Kyle Withers, William H Savran, Peng Wang, and Dawei Mu. 2016. High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers. In SC'16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 957--968.

Digital Library

[30]

USGS M7.6 Shakemap. 2023. M 7.5 - Elbistan earthquake, Kahramanmaras earthquake sequence. https://earthquake.usgs.gov/earthquakes/eventpage/us6000jlqa/shakemap

[31]

USGS M7.8 Shakemap. 2023. M7.8 - Pazarcik earthquake, Kahramanmaras earthquake sequence. https://earthquake.usgs.gov/earthquakes/eventpage/us6000jllz/shakemap

[32]

Patrick Small, David Gill, Philip J Maechling, Ricardo Taborda, Scott Callaghan, Thomas H Jordan, Kim B Olsen, Geoffrey P Ely, and Christine Goulet. 2017. The SCEC unified community velocity model software framework. Seismological Research Letters 88, 6 (2017), 1539--1552.

[33]

Paul Spudich, Margaret Hellweg, and WHK Lee. 1996. Directional topographic site response at Tarzana observed in aftershocks of the 1994 Northridge, California, earthquake: implications for mainshock motions. Bulletin of the Seismological Society of America 86, 1B (1996), S193--S208.

[34]

Jean Virieux. 1984. SH-wave propagation in heterogeneous media: Velocity-stress finite-difference method. Geophysics 49, 11 (1984), 1933--1942.

[35]

Jean Virieux. 1986. P-SV wave propagation in heterogeneous media: Velocity-stress finite-difference method. Geophysics 51, 4 (1986), 889--901.

[36]

Xiaodong Yu, Sheng Di, Kai Zhao, Jiannan Tian, Dingwen Tao, Xin Liang, and Franck Cappello. 2022. Ultrafast Error-bounded Lossy Compression for Scientific Datasets. In Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing. 159--171.

Digital Library

[37]

Wei Zhang and Yang Shen. 2010. Unsplit complex frequency-shifted PML implementation using auxiliary differential equations for seismic wave modeling. Geophysics 75, 4 (2010), T141--T154.

[38]

Wei Zhang, Zhenguo Zhang, and Xiaofei Chen. 2012. Three-dimensional elastic wave numerical modelling in the presence of surface topography by a collocated-grid finite-difference method on curvilinear grids. Geophysical Journal International 190, 1 (2012), 358--378.

[39]

Zhenguo Zhang, Wenqiang Zhang, Danhua Xin, Kejie Chen, and Xiaofei Chen. 2021. A dynamic-rupture model of the 2019 M w 7.1 Ridgecrest earthquake being compatible with the observations. Seismological Research Letters 92, 2A (2021), 870--876.

[40]

Gengshang Zhu, Zhenguo Zhang, Jian Wen, Wei Zhang, and Xiaofei Chen. 2013. Preliminary results of strong ground motion simulation for the Lushan earthquake of 20 April 2013, China. Earthquake Science 26 (2013), 191--197.

Cited By

Li JFeng ZGao YTian SZhang HYe HZhang J(2024)High-Performance 3D convolution on the Latest Generation Sunway ProcessorProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673093(241-251)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673093
Zhang ZWang ZGuo YWang WSun ZWan WGan LHan RWang Y(2024)A Low Overhead Heterogeneous Parallel Optimization Method Based on 3-D Elastic Wave Numerical SimulationIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.339681462(1-11)Online publication date: 2024
https://doi.org/10.1109/TGRS.2024.3396814
Yuan MLiu QGan LYang G(2024)ESFLOW: Mapping Large-Scale Earthquake Simulation to Spatial Computing Systems2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558278(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558278
Show More Cited By

Recommendations

18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLight: enabling depiction of 18-Hz and 8-meter scenarios
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

This paper reports our large-scale nonlinear earthquake simulation software on Sunway TaihuLight. Our innovations include: (1) a customized parallelization scheme that employs the 10 million cores efficiently at both the process and the thread levels; (...
Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquake
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

We present a high-resolution simulation of the 2004 Sumatra-Andaman earthquake, including non-linear frictional failure on a megathrustsplay fault system. Our method exploits unstructured meshes capturing the complicated geometries in subduction zones ...
Towards Efficient SpMV on Sunway Manycore Architectures
ICS '18: Proceedings of the 2018 International Conference on Supercomputing

Sparse Matrix-Vector Multiplication (SpMV) is an essential computation kernel for many data-analytic workloads running in both supercomputers and data centers. The intrinsic irregularity in SpMV is challenging to achieve high performance, especially ...

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences

SC '23: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2023

1428 pages

ISBN:9798400701092

DOI:10.1145/3581784

Chair:
Dorian Arnold,
Program Chair:
Rosa M Badia,
Program Co-chair:
Kathryn Mohror

Copyright © 2023 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

Sponsors

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 11 November 2023

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC '23

Sponsor:

SIGHPC

SC '23: International Conference for High Performance Computing, Networking, Storage and Analysis

November 12 - 17, 2023

CO, Denver, USA

Acceptance Rates

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

6
Total Citations
View Citations
630
Total Downloads

Downloads (Last 12 months)630
Downloads (Last 6 weeks)52

Reflects downloads up to 30 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Li JFeng ZGao YTian SZhang HYe HZhang J(2024)High-Performance 3D convolution on the Latest Generation Sunway ProcessorProceedings of the 53rd International Conference on Parallel Processing10.1145/3673038.3673093(241-251)Online publication date: 12-Aug-2024
https://dl.acm.org/doi/10.1145/3673038.3673093
Zhang ZWang ZGuo YWang WSun ZWan WGan LHan RWang Y(2024)A Low Overhead Heterogeneous Parallel Optimization Method Based on 3-D Elastic Wave Numerical SimulationIEEE Transactions on Geoscience and Remote Sensing10.1109/TGRS.2024.339681462(1-11)Online publication date: 2024
https://doi.org/10.1109/TGRS.2024.3396814
Yuan MLiu QGan LYang G(2024)ESFLOW: Mapping Large-Scale Earthquake Simulation to Spatial Computing Systems2024 IEEE International Symposium on Circuits and Systems (ISCAS)10.1109/ISCAS58744.2024.10558278(1-5)Online publication date: 19-May-2024
https://doi.org/10.1109/ISCAS58744.2024.10558278
Song ZGan LXiang SWang YDuan XYang G(2024)Enabling High-Performance Physical Based Rendering on New Sunway Supercomputer2024 IEEE International Parallel and Distributed Processing Symposium (IPDPS)10.1109/IPDPS57955.2024.00032(277-288)Online publication date: 27-May-2024
https://doi.org/10.1109/IPDPS57955.2024.00032
Wan JWang WZhang Z(2024)Enhancing computational efficiency in 3-D seismic modelling with half-precision floating-point numbers based on the curvilinear grid finite-difference methodGeophysical Journal International10.1093/gji/ggae235238:3(1595-1611)Online publication date: 5-Jul-2024
https://doi.org/10.1093/gji/ggae235
Liu XYin ZTian HWan WHua MZhao WHuang ZGao PZhu FWang HDuan X(2023)Refactoring BZIP2 on the new‐generation sunway supercomputerEngineering Reports10.1002/eng2.12806Online publication date: 3-Nov-2023
https://doi.org/10.1002/eng2.12806

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents