Location via proxy:   [ UP ]  
[Report a bug]   [Manage cookies]                
skip to main content
10.5555/3014904.3015015acmconferencesArticle/Chapter ViewAbstractPublication PagesscConference Proceedingsconference-collections
research-article

High-frequency nonlinear earthquake simulations on petascale heterogeneous supercomputers

Published: 13 November 2016 Publication History

Abstract

The omission of nonlinear effects in large-scale 3D ground motion estimation, which are particularly challenging due to memory and scalability issues, can result in costly misguidance for structural design in earthquake-prone regions. We have implemented nonlinearity using a Drucker-Prager yield condition in AWP-ODC and further optimized the CUDA kernels to more efficiently utilize the GPU's memory bandwidth. The application has resulted in a significant increase in the model region and accuracy for state-of-the-art earthquake simulations in a realistic earth structure, which are now able to resolve the wavefield at frequencies relevant for the most vulnerable buildings (> 1 Hz) while maintaining the scalability and efficiency of the method. We successfully run the code on 4,200 Kepler K20X GPUs on NCSA Blue Waters and OLCF Titan to simulate a M 7.7 earthquake on the southern San Andreas fault with a spatial resolution of 25 m for frequencies up to 4 Hz.

References

[1]
A. Ben-Menahem, "Radiation of seismic surface-waves from finite moving sources", Bull. seism. Soc. Am., vol. 51, no. 3, pp. 401--435, 1961.
[2]
K. B. Olsen, S. M. Day, J. B. Minster, Y. Cui, A. Chourasia, M. Faerman, R. Moore, P. Maechling, and T. Jordan, "TeraShake: Strong shaking in Los Angeles expected from southern San Andreas earthquake", Seism. Res. Lett., vol. 77, pp. 281--282, 2006.
[3]
S. L. Kramer, Geotechnical Earthquake Engineering. Prentice Hall, New Jersey, 1996, p. 653.
[4]
R. Taborda and D. Roten, "Physics-based ground-motion simulation", Encyclopedia of Earthquake Engineering, Springer, Berlin, Heidelberg, pp. 1--33, 2015.
[5]
R. W. Graves, B. T. Aagaard, K. W. Hudnut, L. M. Star, J. P. Stewart, and T. H. Jordan, "Broadband simulations for Mw 7.8 southern San Andreas earthquakes: Ground motion sensitivity to rupture speed", Geophys. Res. Lett., vol. 35, p. L22302, 2008.
[6]
K. B. Olsen, S. M. Day, L. A. Dalguer, J. Mayhew, Y. Cui, J. Zhu, V. Cruz-Atienza, D. Roten, P. Maechling, T. Jordan, D. Okaya, and A. Chourasia, "Shakeout-D: ground motion estimates using an ensemble of large earthquakes on the southern San Andreas fault with spontaneous rupture propagation", Geophys. Res. Lett., vol. 36, p. L04303, 2009.
[7]
Y. Cui, K. B. Olsen, T. H. Jordan, K. Lee, J. Zhou, P. Small, D. Roten, G. Ely, D. K. Panda, A Chourasia, J. Levesque, S. M. Day, and P. Maechling, "Scalable earthquake simulation on petascale supercomputers", in Proceedings of SC10, November 13-19, New Orleans, LA, 2010.
[8]
Southern California Earthquake Center. (2016). Shakeout, {Online}. Available: http://www.shakeout.org.
[9]
J. Zhou, Y. Cui, E. Poyraz, D. J. Choi, and C. C. Guest, "Multi-GPU implementation of a 3D finite difference time domain earthquake code on heterogeneous supercomputers", Procedia Computer Science, vol. 18, pp. 1255--1264, 2013.
[10]
Y. Cui, E. Poyraz, K. B. Olsen, J. Zhou, K. Withers, S. Callaghan, J. Larkin, C. Guest, D Choi, A. Chourasia, et al., "Physics-based seismic hazard analysis on petascale heterogeneous supercomputers", in Proceedings of SC13: International Conference for High Performance Computing, Networking, Storage and Analysis, ACM, 2013, p. 70.
[11]
R. Graves, T. H. Jordan, S. Callaghan, E. Deelman, E. Field, G. Juve, C. Kesselman, P. Maechling, G. Mehta, K. Milner, D. Okaya, P. Small, and K. Vahi, "CyberShake: A physics-based seismic hazard model for southern California", Pure Appl. Geophys., vol. 168, no. 3-4, pp. 367--381, 2011.
[12]
K. Withers, K. B. Olsen, and S. M. Day, "Deterministic high-frequency ground motion using dynamic rupture along rough faults, small-scale media heterogeneities, and frequency-dependent attenuation", in AGU Fall Meeting Abstracts, vol. 1, 2013, p. 08.
[13]
Z.Shi and S. M. Day, "Rupture dynamics and ground motion from 3-D rough-fault simulations", J. Geophys. Res, 2013.
[14]
K. B. Withers, K. B. Olsen, and S. M. Day, "Memory-efficient simulation of frequency-dependent Q", Bull. Seism. Soc. Am., 2015.
[15]
S. M. Day and C. R. Bradley, "Memory-efficient simulation of anelastic wave propagation", Bull. seism. Soc. Am., vol. 91, no. 3, p. 520, 2001.
[16]
E. M. Dunham, D. Belanger, L. Cong, and J. E. Kozdon, "Earthquake ruptures with strongly rate-weakening friction and off-fault plasticity, Part 2: Nonplanar faults", Bull. Seism. Soc. Am., vol. 101, no. 5, pp. 2308--2322, 2011.
[17]
D. Andrews, "Rupture dynamics with energy loss outside the slip zone", J. Geophys. Res, vol. 110, no. B1, B01307, 2005.
[18]
S. Ma and D. J. Andrews, "Inelastic off-fault response and three-dimensional dynamics of earthquake rupture on a strike-slip fault", Journal of Geophysical Research: Solid Earth (1978--2012), vol. 115, no. b4, 2010.
[19]
B. Duan and S. M. Day, "Inelastic strain distribution and seismic radiation from rupture of a fault kink", J. Geophys. Res, vol. 113, no. B12, 2008.
[20]
E. M. Dunham, D. Belanger, L. Cong, and J. E. Kozdon, "Earthquake ruptures with strongly rate-weakening friction and off-fault plasticity, Part 1: planar faults", Bull. Seism. Soc. Am., vol. 101, no. 5, pp. 2296--2307, 2011.
[21]
A.-A. Gabriel, J.-P. Ampuero, L. A. Dalguer, and P. M. Mai, "Source properties of dynamic rupture pulses with off-fault plasticity", J. Geophys. Res., vol. 118, no. 8, pp. 4117--4126, 2013.
[22]
L. F. Bonilla, K. Tsuda, N. Pulido, J. Régnier, and A. Laurendeau, "Nonlinear site response evidence of K-NET and KiK-net records from the 2011 off the Pacific coast of Tohoku Earthquake", Earth Planets and Space, vol. 63, no. 7, p. 785, 2011.
[23]
J. Regnier, L. F. Bonilla, E. Betrand, and J. F. Semblat, "Empirical evidence of nonlinear site response at several KiK-net stations", in 4th IASPEI/IAEE International Symposium: Effects of Surface Geology on Seismic Motion, August 23-26, 2011, University of Santa Barbara, 2011.
[24]
S. Noguchi, H. Sato, and T. Sasatani, "Characterization of nonlinear site response based on strong motion records at K-NET and KiK-net stations in the east of Japan", in Proceedings of the 15th WCEE, Sep 24--28 2012, Lisbon, Portugal, 2012.
[25]
D. Roten, D. Fäh, and L. F. Bonilla, "High-frequency ground motion amplification during the 2011 Tohoku earthquake explained by soil dilatancy", Geophys. J. Int., vol. in press, 2013.
[26]
W. B. Joyner, "Strong Motion from Surface Waves in Deep Sedimentary Basins", Bull. seism. Soc. Am., vol. 90, no. 6B, S95--112, 2000.
[27]
N. H. Sleep, "Nonlinear behavior of strong surface waves trapped in sedimentary basins", Bull. Seism. Soc. Am., vol. 100, no. 2, pp. 826--832, 2010.
[28]
N. H. Sleep and B. A. Erickson, "Nonlinear attenuation of S-waves and Love waves within ambient rock", Geochemistry, Geophysics, Geosystems, vol. 15, no. 4, pp. 1419--1440, 2014.
[29]
D. Roten, K. B. Olsen, S. M. Day, Y. Cui, and D. Fäh, "Expected seismic shaking in Los Angeles reduced by San Andreas fault zone plasticity", Geophysical Research Letters, vol. 41, no. 8, pp. 2769--2777, 2014.
[30]
K. B. Olsen, "Simulation of three-dimensional wave propagation in the Salt Lake basin", PhD thesis, University of Utah, Salt Lake City, Utah, 1994, p. 157.
[31]
C. Cerjan, D. Kosloff, and M. Reshef, "A nonreflecting boundary condition for discrete acoustic and elastic wave equations", Geophysics, vol. 50, pp. 705--708, 1985.
[32]
C. Marcinkovich and K. Olsen, "On the implementation of perfectly matched layers in a three-dimensional fourth-order velocity-stress finite difference scheme", J. geophys. Res, vol. 108, p. 2276, 2003.
[33]
K. C. Meza-Fajardo and A. S. Papageorgiou, "A nonconvolutional, split-field, perfectly matched layer for wave propagation in isotropic and anisotropic elastic media: stability analysis", Bull. seism. Soc. Am., vol. 98, no. 4, p. 1811, 2008, issn: 0037--1106.
[34]
E. Gottschämmer and K. Olsen, "Accuracy of the explicit planar free-surface boundary condition implemented in a fourth-order staggered-grid velocity-stress finite-difference scheme", Bull. seism. Soc. Am., vol. 91, no. 3, p. 617, 2001.
[35]
S. M. Day, "Efficient simulation of constant Q using coarse-grained memory variables", Bull. seism. Soc. Am., vol. 88, no. 4, p. 1051, 1998.
[36]
R. Harris, M. Barall, R. Archuleta, E. Dunham, B. Aagaard, J. Ampuero, H. Bhat, V. Cruz-Atienza, L. Dalguer, P. Dawson, et al., "The SCEC/USGS dynamic earthquake rupture code verification exercise", Seismological Research Letters, vol. 80, no. 1, pp. 119--126, 2009.
[37]
L. A. Dalguer and S. M. Day, "Staggered-grid split-node method for spontaneous rupture simulation", J. Geoph. Res., vol. 112, B02302, 2007.
[38]
R. A. Harris, M. Barall, D. J. Andrews, B. Duan, S. Ma, E. M. Dunham, A.-A. Gabriel, Y. Kaneko, Y. Kase, B. T. Aagaard, J.-P. Oglesby, J. P. Ampuero, T. C. Hanks, and N. Abrahamson, "Verifying a computational method for predicting extreme ground motion", Seism. Res. Lett., vol. 82, no. 5, pp. 638--644, 2011.
[39]
Barall, M. (2016). The SCEC/USGS Spontaneous Rupture Code Verification Project, {Online}. Available: http://scecdata.usc.edu/cvws/.
[40]
Y. Cui, R. Moore, K. Olsen, A. Chourasia, P. Maechling, B. Minster, S. Day, Y. Hu, J. Zhu, and T. Jordan, "Toward petascale earthquake simulations", Acta Geotechnica, vol. 4, no. 2, pp. 79--93, 2009.
[41]
J. Zhou, D. Unat, D. J. Choi, C. C. Guest, and Y. Cui, "Hands-on performance tuning of 3D finite difference earthquake simulation on GPU Fermi chipset", Procedia Computer Science, vol. 9, pp. 976--985, 2012.
[42]
J. Yu, D. Roten, and Y. Cui, "Development of Parallel IO Interface for High Performance SEISM-IO Library", in SCEC 2015 Annual Meeting proceeding and abstracts, 2015.
[43]
H. Xu, E. Poyraz, D. Roten, and Y. Cui, "A High Level Parallel IO Library for High- Performance Seismic Applications", in SCEC 2014 Annual Meeting proceeding and abstracts, 2014.
[44]
The HDF Group. (1997-2016). Hierarchical Data Format, version 5, {Online}. Available: http://www.hdfgroup.org/HDF5/.
[45]
J. F. Lofstead, S. Klasky, K. Schwan, N. Podhorszki, and C. Jin, "Flexible IO and integration for scientific codes through the adaptable IO system (ADIOS)", in Proceedings of the 6th international workshop on Challenges of large applications in distributed environments, ACM, 2008, pp. 15--24.
[46]
J. Li, W.-K. Liao, A. Choudhary, R. Ross, R. Thakur, W. Gropp, R. Latham, A. Siegel, B. Gallagher, and M. Zingale, "Parallel netCDF: a high-performance scientific I/O interface", in Supercomputing, 2003 ACM/IEEE Conference, IEEE, 2003, pp. 39--39.
[47]
M. L. Wilkins, "Calculation of elastic-plastic flow", in Methods of Computational Physics, B. Alder, Ed. Academic Press, New York, 1964, vol. 3.
[48]
J. C. Simo and T. J. R. Hughes, "Computational inelasticity", in Interdisciplinary Applied Mathematics, J. E. Marsden, L. Sirovich, and 5. Wiggins, Eds. Springer-Verlag, Berlin, 1998, vol. 7.
[49]
D. Andrews, T. C. Hanks, and J. W. Whitney, "Physical limits on ground motion at Yucca Mountain", Bull. seism. Soc. Am., vol. 97, no. 6, pp. 1771--1792, 2007.
[50]
R. Madariaga, "Dynamics of an expanding circular fault", Bull. Seism. Soc. Am., vol. 66, no. 3, pp. 639--666, 1976.
[51]
D. Roten, K. B. Olsen, Y. Cui, and S. M. Day, "Quantification of fault zone plasticity effects with spontaneous rupture simulations", in Best Practices in Physics-based Fault Rupture Models for Seismic Hazard Assessment of Nuclear Installations Vienna, Austria, November 18-20, 2015, 2015.
[52]
Y. Cui, E. Poyraz, J. Zhou, S. Callaghan, P. Maechling, T. H. Jordan, L. Shih, and P. Chen, "Accelerating Cybershake calculations on the XE6/XK7 platform of Blue Waters", in Extreme Scaling Workshop (XSW), 2013, IEEE, 2013, pp. 8--17.
[53]
R. Weldon, K. Scharer, T. Fumal, and G. Biasi, "Wrightwood and the earthquake cycle: what a long recurrence record tells us about how faults work", Geol. Seism. Am. Today, vol. 14, pp. 4--10, 2004.
[54]
S. M. Day, D. Roten, and K. B. Olsen, "Adjoint analysis of the source and path sensitivities of basin-guided waves", Geophys. J. Int., vol. 189, no. 2, pp. 1103--1124, 2012.
[55]
S. Hartzell, S. Harmsen, and A. Frankel, "Effects of 3D random correlated velocity perturbations on predicted ground motions", Bull. Seism. Soc. Am., vol. 100, no. 4, pp. 1415--1426, 2010.
[56]
W. Imperatori and P. M. Mai, "Broad-band near-field ground motion simulations in 3-dimensional scattering media", Geophysical Journal International, vol. 192, no. 2, pp. 725--744, 2013.
[57]
W. Imperatori and P. M. Mai, "The role of topography and lateral velocity heterogeneities on near-source scattering and ground-motion variability", Geophysical Journal International, vol. 202, no. 3, pp. 2163--2181, 2015.
[58]
H. Magistrale, S. M Day, R. W. Clayton, and R. Graves, "The SCEC Southern California Reference Three-Dimensional Seismic Velocity Model Version 2", Bull. Seism. Soc. Am., vol. 90, no. 6B, S65--S76, 2000.
[59]
E.-J. Lee, P. Chen, and T. H. Jordan, "Testing waveform predictions of 3D velocity models against two recent Los Angeles earthquakes", Seismological Research Letters, vol. 85, no. 6, pp. 1275--1284, 2014.
[60]
N. Nakata and G. C. Beroza, "Stochastic characterization of mesoscale seismic velocity heterogeneity in Long Beach, California", Geophysical Journal International, vol. 203, no. 3, pp. 2049--2054, 2015.
[61]
W. H. Savran and K. B. Olsen, "Model for small-scale crustal heterogeneity in Los Angeles basin based on inversion of sonic log data", Geophysical Journal International, vol. 205, no. 2, pp. 856--863, 2016.
[62]
L. Klimeš, "Correlation functions of random media", Pure and Applied Geophysics, vol. 159, no. 7-8, pp. 1811--1831, 2002.
[63]
K. Withers, K. B. Olsen, Z.Shi, and S. Day, "Broadband (0-8 Hz) Ground Motion Variability from Ensemble Simulations of Buried Mw 6.7 Thrust Earthquakes Including Rough Fault Descriptions and Q(f)", in SCEC 2015 Annual Meeting proceeding and abstracts, 2015.
[64]
E. Hoek, C. Carranza-Torres, and B. Corkum, "Hoek-Brown failure criterion - 2002 edition", Proceedings of NARMS-Tac, vol. 1, pp. 267--273, 2002.
[65]
L. M. Flesch, W. E. Holt, A. J. Haines, and B. Shen-Tu, "Dynamics of the Pacific-North American plate boundary in the western United States", Science, vol. 287, no. 5454, pp. 834--836, 2000.
[66]
C. H. Scholz, "Evidence for a strong San Andreas fault", Geology, vol. 28, no. 2, 2002.
[67]
L. A. Dalguer, S. M. Day, K. B. Olsen, and V. M. Cruz-Atienza, "Rupture models and ground motion for Shakeout and other southern San Andreas fault scenarios", in Proceedings of 14th World Conference on Earthquake Engineering, Int. Assoc. for Earthquake Eng., Beijing, China, 2008.
[68]
D. Roten, K. B. Olsen, J. C. Pechmann, V. M. Cruz-Atienza, and H. Magistrale, "3D simulations of M 7 earthquakes on the Wasatch fault, Utah, Part I: Long-period (0-1 Hz) ground motions", Bull. seism. Soc. Am., vol. 101, no. 5, pp. 2045--2063, 2011.
[69]
L. A. Dalguer and P. M. Mai, "Prediction of near-source ground motion exceeding 1g at low frequencies (< 2Hz) from Mw 6.5 deterministic and physics-based dynamic rupture simulations", in Proceedings of 15th World Conference on Earthquake Engineering, Int. Assoc. for Earthquake Eng., Lisbon, Portugal, 2012.
[70]
C. Baumann and L. A. Dalguer, "Evaluating the compatibility of dynamic rupture-based synthetic ground motion with empirical ground-motion prediction equation", Bull. Seism. Soc. Am., vol. 104, no. 2, 2014.
[71]
P. M. Mai and G. C. Beroza, "A spatial random field model to characterize complexity in earthquake slip", Journal of Geophysical Research (Solid Earth), vol. 107, pp. 2308--+, Nov. 2002.
[72]
R. W. Graves, B. T. Aagaard, and K. W. Hudnut, "The Shakeout earthquake source and ground motion simulations", Earthquake Spectra, vol. 27, no. 2, pp. 273--291, 2011.
[73]
D. Andrews, "Test of two methods for faulting in finite-difference calculations", Bull. Seism. Soc. Am., vol. 89, no. 4, pp. 931--937, 1999.
[74]
S. Nie, "Discontinuous staggered finite difference method for anelastic wave simulations", in 2015 AGU Fall Meeting, AGU, 2015.

Cited By

View all
  • (2023)69.7-PFlops Extreme Scale Earthquake Simulation with Crossing Multi-faults and Topography on SunwayProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613209(1-15)Online publication date: 12-Nov-2023
  • (2021)3D acoustic-elastic coupling with gravityProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476173(1-14)Online publication date: 14-Nov-2021
  • (2019)Automatic, application-aware I/O forwarding resource allocationProceedings of the 17th USENIX Conference on File and Storage Technologies10.5555/3323298.3323323(265-279)Online publication date: 25-Feb-2019
  • Show More Cited By

Recommendations

Comments

Information & Contributors

Information

Published In

cover image ACM Conferences
SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis
November 2016
1034 pages
ISBN:9781467388153
  • Conference Chair:
  • John West

Sponsors

In-Cooperation

Publisher

IEEE Press

Publication History

Published: 13 November 2016

Check for updates

Author Tags

  1. GPU
  2. SCEC
  3. earthquake ground motion
  4. fault zone plasticity
  5. nonlinear soil behavior

Qualifiers

  • Research-article

Conference

SC16
Sponsor:

Acceptance Rates

SC '16 Paper Acceptance Rate 81 of 442 submissions, 18%;
Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)3
  • Downloads (Last 6 weeks)0
Reflects downloads up to 30 Aug 2024

Other Metrics

Citations

Cited By

View all
  • (2023)69.7-PFlops Extreme Scale Earthquake Simulation with Crossing Multi-faults and Topography on SunwayProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3581784.3613209(1-15)Online publication date: 12-Nov-2023
  • (2021)3D acoustic-elastic coupling with gravityProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3458817.3476173(1-14)Online publication date: 14-Nov-2021
  • (2019)Automatic, application-aware I/O forwarding resource allocationProceedings of the 17th USENIX Conference on File and Storage Technologies10.5555/3323298.3323323(265-279)Online publication date: 25-Feb-2019
  • (2019)In-situ Analysis and Visualization of Earthquake SimulationPractice and Experience in Advanced Research Computing 2019: Rise of the Machines (learning)10.1145/3332186.3332201(1-5)Online publication date: 28-Jul-2019
  • (2019)Characterizing CUDA Unified Memory (UM)-Aware MPI Designs on Modern GPU ArchitecturesProceedings of the 12th Workshop on General Purpose Processing Using GPUs10.1145/3300053.3319419(43-52)Online publication date: 13-Apr-2019
  • (2018)Vectorization of a spectral finite-element numerical kernelProceedings of the 2018 4th Workshop on Programming Models for SIMD/Vector Processing10.1145/3178433.3178441(1-7)Online publication date: 24-Feb-2018
  • (2018)An algorithm for fast elastic wave simulation using a vectorized finite difference operatorComputers & Geosciences10.1016/j.cageo.2018.04.002116:C(23-31)Online publication date: 1-Jul-2018
  • (2017)Extreme scale multi-physics simulations of the tsunamigenic 2004 sumatra megathrust earthquakeProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3126908.3126948(1-16)Online publication date: 12-Nov-2017
  • (2017)18.9-Pflops nonlinear earthquake simulation on Sunway TaihuLightProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.1145/3126908.3126910(1-12)Online publication date: 12-Nov-2017
  • (2017)Fast and Scalable Low-Order Implicit Unstructured Finite-Element Solver for Earth's Crust Deformation ProblemProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3093172.3093236(1-10)Online publication date: 26-Jun-2017

View Options

Get Access

Login options

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media