research-article

Open access

Performance Analysis and Optimization of Nonhydrostatic ICosahedral Atmospheric Model (NICAM) on the K Computer and TSUBAME2.5

Authors:

Hisashi Yashiro,

Hirofumi TomitaAuthors Info & Claims

PASC '16: Proceedings of the Platform for Advanced Scientific Computing Conference

Article No.: 3, Pages 1 - 8

https://doi.org/10.1145/2929908.2929911

Published: 08 June 2016 Publication History

Abstract

We summarize the optimization and performance evaluation of the Nonhydrostatic ICosahedral Atmospheric Model (NICAM) on two different types of supercomputers: the K computer and TSUBAME2.5. First, we evaluated and improved several kernels extracted from the model code on the K computer. We did not significantly change the loop and data ordering for sufficient usage of the features of the K computer, such as the hardware-aided thread barrier mechanism and the relatively high bandwidth of the memory, i.e., a 0.5 Byte/FLOP ratio. Loop optimizations and code cleaning for a reduction in memory transfer contributed to a speed-up of the model execution time. The sustained performance ratio of the main loop of the NICAM reached 0.87 PFLOPS with 81,920 nodes on the K computer. For GPU-based calculations, we applied OpenACC to the dynamical core of NICAM. The performance and scalability were evaluated using the TSUBAME2.5 supercomputer. We achieved good performance results, which showed efficient use of the memory throughput performance of the GPU as well as good weak scalability. A dry dynamical core experiment was carried out using 2560 GPUs, which achieved 60 TFLOPS of sustained performance.

References

[1]

T. Arakawa, T. Inoue, and M. Sato. Performance evaluation and case study of a coupling software ppOpen-MATH/MP. Procedia Comput. Sci., 29:924--935, 2014.

[2]

T. Arakawa, H. Yoshimura, F. Saito, and K. Ogochi. Data exchange algorithm and software design of KAKUSHIN coupler Jcup. Procedia Comput. Sci., 4:1516--1525, 2011.

[3]

K. Aranami, T. Hara, Y. Ikuta, K. Kawano, K. Matsubayashi, H. Kusabiraki, T. Ito, T. Egawa, K. Yamashita, Y. Ota, Y. Ishikawa, T. Fujita, and J. Ishida. A new operational regional model for convection-permitting numerical weather prediction at JMA. CAS/JSC WGNE Research Activities in Atmospheric and Oceanic Modelling, 2015.

[4]

O. Fuhrer, C. Osuna, X. Lapillonne, T. Gysi, B. Cumming, M. Bianco, A. Arteaga, and T. C. Schulthess. Towards a performance portable, architecture agnostic implementation strategy for weather and climate models. Supercomput. Frontiers Innov., 1(1):45--62, 2014.

Digital Library

[5]

W. W. Grabowski. Toward cloud resolving modeling of large-scale tropical circulations: A simple cloud microphysics parameterization. J. Atmos. Sci., 55:3283--3298, 1998.

[6]

H. Hasumi. CCSR ocean component model (COCO). CCSR Rep 13. The University of Tokyo, Chiba, Japan, 2000.

[7]

H. Hasumi. CCSR ocean component model (COCO), version 4.0. CCSR Rep 25. The University of Tokyo, Chiba, Japan, 2006.

[8]

J. Ishida, C. Muroi, and K. Kawano. Development of a new nonhydrostatic model ASUCA at JMA. CAS/JSC WGNE Research Activities in Atmospheric and Oceanic Modelling, 2010.

[9]

C. Jablonowski and D. L. Williamson. A baroclinic instability test case for atmospheric model dynamical cores. Q. J. R. Meteorol. Soc., 132(621C):2943--2975, 2007.

[10]

J. B. Klemp and R. B. Wilhelmson. The simulation of three-dimensional convective storm dynamics. J. Atmos. Sci., 35(6):1070--1096, 1978.

[11]

C. Kodama, Y. Yamada, A. T. Noda, K. Kikuchi, Y. Kajikawa, T. Nasuno, T. Tomita, T. Yamamura, H. G. Takahashi, M. Hara, Y. Kawatani, and M. Satoh. A 20-year climatology of a NICAM AMIP-type simulation. J. Meteorol. Soc. Japan, 93(4):393--424, 2015.

[12]

Y. Matsuzaki, N. Uchikoga, M. Ohue, T. Shimoda, T. Sato, T. Ishida, and Y. Akiyama. MEGADOCK 3.0: A high-performance protein-protein interaction prediction software using hybrid parallel computing for petascale supercomputing environments. Source Code Biol. Med., 8(1):18, 2013.

[13]

H. Miura, M. Satoh, T. Nasuno, A. T. Noda, and K. Oouchi. A Madden-Julian oscillation event realistically simulated by a global cloud-resolving model. Science, 318(5857):1763--1765, 2007.

[14]

Y. Miyamoto, Y. Kajikawa, R. Yoshida, T. Yamaura, H. Yashiro, and H. Tomita. Deep moist atmospheric convection in a subkilometer global simulation. Geophys. Res. Lett., 40(18):4922--4926, 2013.

[15]

M. Nakanishi and H. Niino. An improved Mellor--Yamada level-3 model: Its numerical stability and application to a regional prediction of advection fog. Bound.-Layer Meteorol., 119(2):397--407, 2006.

[16]

M. Nakanishi and H. Niino. Development of an improved turbulence closure model for the atmospheric boundary layer. J. Meteorol. Soc. Japan, 87(5):895--912, 2009.

[17]

A. T. Noda, K. Oouchi, M. Satoh, H. Tomita, S. Iga, and Y. Tsushima. Importance of the subgrid-scale turbulent moist process: Cloud distribution in global cloud-resolving simulations. Atmos. Res., 96(2--3):208--217, 2010.

[18]

M. Satoh. Conservative scheme for the compressible nonhydrostatic models with the horizontally explicit and vertically implicit time integration scheme. Mon. Wea. Rev., 130(5):1227--1245, 2002.

[19]

M. Satoh. Conservative scheme for a compressible nonhydrostatic model with moist processes. Mon. Wea. Rev., 131(6):1033--1050, 2003.

[20]

M. Satoh, T. Matsuno, H. Tomita, H. Miura, T. Nasuno, and S. Iga. Nonhydrostatic icosahedral atmospheric model (NICAM) for global cloud resolving simulations. J. Comput. Phys., 227(7):3486--3514, 2008.

Digital Library

[21]

M. Satoh, H. Tomita, H. Yashito, H. Miura, C. Kodama, T. Seiki, A. T. Noda, Y. Yamada, D. Goto, M. Sawada, T. Miyoshi, Y. Niwa, M. Hara, T. Ohno, S. Iga, T. Arakawa, T. Inoue, and H. Kubokawa. The Non-hydrostatic Icosahedral Atmospheric Model: Description and development. Prog. Earth Planet. Sci., 1(1):18, 2014.

[22]

T. Seiki and T. Nakajima. Aerosol effects of the condensation process on a convective cloud simulation. J. Atmos. Sci., 71(2):833--853, 2014.

[23]

T. Shimokawabe, T. Aoki, J. Ishida, K. Kawano, and C. Muroi. 145 TFlops performance on 3990 GPUs of TSUBAME 2.0 supercomputer for an operational weather prediction. Procedia Comput. Sci., 4:1535--1544, 2011.

[24]

T. Shimokawabe, T. Aoki, and N. Onodera. Performance optimization and evaluation of a global climate application using a 440m horizontal mesh on the K computer. In SC14: The International Conference for High Performance Computing, Networking, Storage and Analysis, pages 251--261. IEEE, November 2014.

Digital Library

[25]

M. Terai, H. Yashiro, K. Sakamoto, S. Iga, and H. Tomita. High-productivity framework on GPU-rich supercomputers for operational weather prediction code ASUCA. In SC14: The International Conference for High Performance Computing, Networking, Storage and Analysis, November 2014.

Digital Library

[26]

K. Terasaki, M. Sawada, and T. Miyoshi. Local ensemble transform Kalman filter experiments with the Nonhydrostatic Icosahedral Atmospheric Model NICAM. SOLA, 11:23--26, 2015.

[27]

H. Tomita. New microphysical schemes with five and six categories by diagnostic generation of cloud ice. J. Meteorol. Soc. Japan, 86A:121--142, 2008.

[28]

H. Tomita, K. Goto, and M. Satoh. A new approach to atmospheric general circulation model: Global cloud resolving model NICAM and its computational performance. SIAM J. Sci. Comput., 30(6):2755--2776, 2008.

Digital Library

[29]

H. Tomita, H. Miura, S. Iga, T. Nasuno, and M. Satoh. A global cloud-resolving simulation: Preliminary results from an aqua planet experiment. Geophys. Res. Lett., 32(8):L08805, 2005.

[30]

H. Tomita and M. Satoh. A new dynamical framework of nonhydrostatic global model using the icosahedral grid. Fluid Dyn. Res., 34(6):357--400, 2004.

[31]

H. Tomita, M. Satoh, and K. Goto. An optimization of the icosahedral grid modified by spring dynamics. J. Comput. Phys., 183(1):307--331, 2002.

Digital Library

[32]

H. Tomita, M. Tsugawa, M. Satoh, and K. Goto. Shallow water model on a modified icosahedral geodesic grid by using spring dynamics. J. Comput. Phys., 174(2):579--613, 2001.

Digital Library

[33]

H. Yashiro, K. Terasaki, T. Miyoshi, and H. Tomita. Performance evaluation of throughput-aware framework for ensemble data assimilation: The case of NICAM-LETKF. Geosci. Model Dev. Discuss., submitted.

Cited By

Noda AOhno TKodama CChen YKuba NSeiki TYamada YSatoh M(2023)Recent global nonhydrostatic modeling approach without using a cumulus parameterization to understand the mechanisms underlying cloud changes due to global warmingProgress in Earth and Planetary Science10.1186/s40645-023-00583-x10:1Online publication date: 18-Aug-2023
https://doi.org/10.1186/s40645-023-00583-x
Jin YWang HZhong RZhang CZhai JLee JAgrawal KSpear M(2022)PerFlowProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming10.1145/3503221.3508405(177-191)Online publication date: 2-Apr-2022
https://dl.acm.org/doi/10.1145/3503221.3508405
Kodama COhno TSeiki TYashiro HNoda ANakano MYamada YRoh WSatoh MNitta TGoto DMiura HNasuno TMiyakawa TChen YSugi M(2021)The Nonhydrostatic ICosahedral Atmospheric Model for CMIP6 HighResMIP simulations (NICAM16-S): experimental design, model description, and impacts of model updatesGeoscientific Model Development10.5194/gmd-14-795-202114:2(795-820)Online publication date: 4-Feb-2021
https://doi.org/10.5194/gmd-14-795-2021
Show More Cited By

Performance Analysis and Optimization of Nonhydrostatic ICosahedral Atmospheric Model (NICAM) on the K Computer and TSUBAME2.5
1. Applied computing
  1. Physical sciences and engineering
2. General and reference
  1. Cross-computing tools and techniques

Recommendations

The CLAW DSL: Abstractions for Performance Portable Weather and Climate Models
PASC '18: Proceedings of the Platform for Advanced Scientific Computing Conference

In order to profit from emerging high-performance computing systems, weather and climate models need to be adapted to run efficiently on different hardware architectures such as accelerators. This is a major challenge for existing community models that ...
Acceleration of a Meteorological Limited Area Model with Dataflow Engines
SAAHPC '12: Proceedings of the 2012 Symposium on Application Accelerators in High Performance Computing

Climate and weather modeling is a significant consumer of High Performance Computing due to the hard deadlines inherent in predicting weather. Given the large data volumes and runtimes involved, climate and weather modeling is ideally suited for ...
Multi-GPU implementation of the NICAM atmospheric model
Euro-Par'12: Proceedings of the 18th international conference on Parallel processing workshops

Climate simulation models are used for a variety of scientific problems and accuracy of the climate prognoses is mostly limited by the resolution of the models. Finer resolution results in more accurate prognoses but, at the same time, significantly ...

Comments

Information & Contributors

Information

Published In

cover image ACM Other conferences

PASC '16: Proceedings of the Platform for Advanced Scientific Computing Conference

June 2016

141 pages

ISBN:9781450341264

DOI:10.1145/2929908

Copyright © 2016 ACM.

Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than the author(s) must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

In-Cooperation

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing
ETH Zurich
CSCS: Swiss National Supercomputing Centre

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 08 June 2016

Permissions

Request permissions for this article.

Request Permissions

Check for updates

Author Tags

Qualifiers

Research-article
Research
Refereed limited

Conference

PASC '16

PASC '16: Platform for Advanced Scientific Computing Conference

June 8 - 10, 2016

Lausanne, Switzerland

Acceptance Rates

PASC '16 Paper Acceptance Rate 12 of 44 submissions, 27%;

Overall Acceptance Rate 109 of 221 submissions, 49%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

15
Total Citations
View Citations
484
Total Downloads

Downloads (Last 12 months)47
Downloads (Last 6 weeks)7

Reflects downloads up to 18 Aug 2024

Other Metrics

View Author Metrics

Citations

Cited By

Noda AOhno TKodama CChen YKuba NSeiki TYamada YSatoh M(2023)Recent global nonhydrostatic modeling approach without using a cumulus parameterization to understand the mechanisms underlying cloud changes due to global warmingProgress in Earth and Planetary Science10.1186/s40645-023-00583-x10:1Online publication date: 18-Aug-2023
https://doi.org/10.1186/s40645-023-00583-x
Jin YWang HZhong RZhang CZhai JLee JAgrawal KSpear M(2022)PerFlowProceedings of the 27th ACM SIGPLAN Symposium on Principles and Practice of Parallel Programming10.1145/3503221.3508405(177-191)Online publication date: 2-Apr-2022
https://dl.acm.org/doi/10.1145/3503221.3508405
Kodama COhno TSeiki TYashiro HNoda ANakano MYamada YRoh WSatoh MNitta TGoto DMiura HNasuno TMiyakawa TChen YSugi M(2021)The Nonhydrostatic ICosahedral Atmospheric Model for CMIP6 HighResMIP simulations (NICAM16-S): experimental design, model description, and impacts of model updatesGeoscientific Model Development10.5194/gmd-14-795-202114:2(795-820)Online publication date: 4-Feb-2021
https://doi.org/10.5194/gmd-14-795-2021
Klingmüller KLelieveld J(2021)Climate-model-informed deep learning of global soil moisture distributionGeoscientific Model Development10.5194/gmd-14-4429-202114:7(4429-4441)Online publication date: 19-Jul-2021
https://doi.org/10.5194/gmd-14-4429-2021
Wang PJiang JLin PDing MWei JZhang FZhao LLi YYu ZZheng WYu YChi XLiu H(2021)The GPU version of LASG/IAP Climate System Ocean Model version 3 (LICOM3) under the heterogeneous-compute interface for portability (HIP) framework and its large-scale applicationGeoscientific Model Development10.5194/gmd-14-2781-202114:5(2781-2799)Online publication date: 18-May-2021
https://doi.org/10.5194/gmd-14-2781-2021
Bertagna LGuba OTaylor MFoucar JLarkin JBradley ARajamanickam SSalinger ACuicchi CQualters IKramer W(2020)A performance-portable nonhydrostatic atmospheric dycore for the energy exascale earth system model running at cloud-resolving resolutionsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3433701.3433823(1-14)Online publication date: 9-Nov-2020
https://dl.acm.org/doi/10.5555/3433701.3433823
Yashiro HTerasaki KKawai YKudo SMiyoshi TImamura TMinami KInoue HNishiki TSaji TSatoh MTomita HCuicchi CQualters IKramer W(2020)A 1024-member ensemble data assimilation with 3.5-km mesh global weather simulationsProceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis10.5555/3433701.3433703(1-10)Online publication date: 9-Nov-2020
https://dl.acm.org/doi/10.5555/3433701.3433703
Bertagna LGuba OTaylor MFoucar JLarkin JBradley ARajamanickam SSalinger A(2020)A Performance-Portable Nonhydrostatic Atmospheric Dycore for the Energy Exascale Earth System Model Running at Cloud-Resolving Resolutions.SC20: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41405.2020.00096(1-14)Online publication date: Nov-2020
https://doi.org/10.1109/SC41405.2020.00096
Yashiro HTerasaki KKawai YKudo SMiyoshi TImamura TMinami KInoue HNishiki TSaji TSatoh MTomita H(2020)A 1024-Member Ensemble Data Assimilation with 3.5-Km Mesh Global Weather SimulationsSC20: International Conference for High Performance Computing, Networking, Storage and Analysis10.1109/SC41405.2020.00005(1-10)Online publication date: Nov-2020
https://doi.org/10.1109/SC41405.2020.00005
Hatfield SChantry MDüben PPalmer T(2019)Accelerating High-Resolution Weather Models with Deep-Learning HardwareProceedings of the Platform for Advanced Scientific Computing Conference10.1145/3324989.3325711(1-11)Online publication date: 12-Jun-2019
https://dl.acm.org/doi/10.1145/3324989.3325711
Show More Cited By

View Options

View options

PDF

View or Download as a PDF file.

eReader

View online with eReader.

Get Access

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

Media

Figures

Other

Tables

View Table of Contents