research-article

Refactoring and optimizing the community atmosphere model (CAM) on the sunway taihulight supercomputer

SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

Article No.: 83, Pages 1 - 12

Published: 13 November 2016 Publication History

Abstract

This paper reports our efforts on refactoring and optimizing the Community Atmosphere Model (CAM) on the Sunway TaihuLight supercomputer, which uses a many-core processor that consists of management processing elements (MPEs) and clusters of computing processing elements (CPEs). To map the large code base of CAM to the millions of cores on the Sunway system, we take OpenACC-based refactoring as the major approach, and apply source-to-source translator tools to exploit the most suitable parallelism for the CPE cluster, and to fit the intermediate variable into the limited on-chip fast buffer. For individual kernels, when comparing the original ported version using only MPEs and the refactored version using both the MPE and CPE clusters, we achieve up to 22x speedup for the compute-intensive kernels. For the 25km resolution CAM global model, we manage to scale to 24,000 MPEs, and 1,536,000 CPEs, and achieve a simulation speed of 2.81 model years per day.

References

[1]

J. Drake, I. Foster, J. Michalakes, B. Toonen, and P. Worley, "Design and Performance of a Scalable Parallel Community Climate Model," Parallel Computing, pp. 1571--1591, 1995.

Digital Library

Google Scholar

[2]

S. Shingu, H. Takahara, H. Fuchigami, M. Y. Y. Tsuda, M. Yamada, Y. Tsuda, and et. al., "A 26.58 Tflops Global Atmospheric Simulation with the Spectral Transform Method on the Earth Simulator," in In Proceedings of the ACM / IEEE Supercomputing SC2002 conference, 2002.

Digital Library

Google Scholar

[3]

X.-J. Yang, X.-K. Liao, K. Lu, Q.-F. Hu, J.-Q. Song, and J.-S. Su, "The TianHe-1A Supercomputer: Its Hardware and Software," Journal of Computer Science and Technology, vol. 26, no. 3, pp. 344--351, 2011.

Crossref

Google Scholar

[4]

X. Liao, L. Xiao, C. Yang, and Y. Lu, "MilkyWay-2 super-computer: system and application," Frontiers of Computer Science, vol. 8, no. 3, pp. 345--356, 2014.

Digital Library

Google Scholar

[5]

J. Michalakes and M. Vachharajani, "GPU acceleration of numerical weather prediction," in Parallel and Distributed Processing, 2008. IPDPS 2008. IEEE International Symposium on, April 2008, pp. 1--7.

Google Scholar

[6]

R. Kelly, "GPU Computing for Atmospheric Modeling," Computing in Science and Engineering, vol. 12, no. 4, pp. 26--33, 2010.

Digital Library

Google Scholar

[7]

Z. Wang, X. Xu, N. Xiong, L. Yang, and W. Zhao, "GPU Acceleration for GRAPES Meteorological Model," in High Performance Computing and Communications (HPCC), 2011 IEEE 13th International Conference on, Sept 2011, pp. 365--372.

Digital Library

Google Scholar

[8]

I. Carpenter, R. Archibald, K. Evans, J. Larkin, P. Micikevicius, M. Norman, and et. al., "Progress towards accelerating HOMME on hybrid multi-core systems," International Journal of High Performance Computing Applications, vol. 27, no. 3, pp. 335--347, 2013.

Digital Library

Google Scholar

[9]

T. Shimokawabe, T. Aoki, C. Muroi, J. Ishida, K. Kawano, T. Endo, and et. al., "An 80-Fold Speedup, 15.0 TFlops Full GPU Acceleration of Non-Hydrostatic Weather Model ASUCA Production Code," in High Performance Computing, Networking, Storage and Analysis (SC), 2010 International Conference for, Nov 2010, pp. 1--11.

Digital Library

Google Scholar

[10]

S. Xu, X. Huang, Y. Zhang, H. Fu, L.-Y. Oey, F. Xu, and et. al., "gpuPOM: a GPU-based Princeton Ocean Model," Geoscientific Model Development Discussions, vol. 7, no. 6, pp. 7651--7691, 2014.

Google Scholar

[11]

M. Govett, J. Middlecoff, and T. Henderson, "Directive-Based Parallelization of the NIM Weather Model for GPUs," in Accelerator Programming using Directives (WACCPD), 2014 First Workshop on, Nov 2014, pp. 55--61.

Digital Library

Google Scholar

[12]

B. Cumming, C. Osuna, T. Gysi, M. Bianco, X. Lapillonne, O. Fuhrer, and T. C. Schulthess, "A review of the challenges and results of refactoring the community climate code COSMO for hybrid Cray HPC systems," Proceedings of Cray User Group, 2013.

Google Scholar

[13]

J. M. Dennis, M. Vertenstein, P. H. Worley, A. A. Mirin, A. P. Craig, R. Jacob, and et. al., "Computational performance of ultra-high-resolution capability in the Community Earth System Model," International Journal of High Performance Computing Applications, vol. 26, no. 1, pp. 5--16, 2012.

Digital Library

Google Scholar

[14]

R. B. Neale and et al., "Description of the NCAR Community Atmosphere Model (CAM 5.0)," Natl.Cent. for Atmos. Res., Boulder, Colo., Tech. Rep. Note NCAR/TN-4861STR.

Google Scholar

[15]

J. M. Dennis, J. Edwards, K. J. Evans, O. Guba, P. H. Lauritzen, A. A. Mirin, and et. al., "CAM-SE: A scalable spectral element dynamical core for the Community Atmosphere Model," International Journal of High Performance Computing Applications, vol. 26, no. 1, pp. 74--89, 2012.

Digital Library

Google Scholar

[16]

J. Linford, J. Michalakes, M. Vachharajani, and A. Sandu, "Multi-core acceleration of chemical kinetics for simulation and prediction," in High Performance Computing Networking, Storage and Analysis, Proceedings of the Conference on, Nov 2009, pp. 1--11.

Digital Library

Google Scholar

[17]

H. Fu, J. Liao, J. Yang, L. Wang, Z.Song, X. Huang, C. Yang, W. Xue, F. Liu, F. Qiao, W. Zhao, X. Yin, C. Hou, C. Zhang, W. Ge, J. Zhang, Y. Wang, C. Zhou, and G. Yang, "The sunway taihulight supercomputer: system and applications," Science China Information Sciences, pp. 1--16, 2016. {Online}. Available: http://dx.doi.org/10.1007/s11432-016-5588-7

Crossref

Google Scholar

[18]

R. J. Small, J. Bacmeister, D. Bailey, A. Baker, S. Bishop, F. Bryan, J. Caron, J. Dennis, P. Gent, H.-m. Hsu, M. Jochum, D. Lawrence, E. Muoz, P. diNezio, T. Scheitlin, R. Tomas, J. Tribbia, Y.-h. Tseng, and M. Vertenstein, "A new synoptic scale resolving global climate simulation using the community earth system model," Journal of Advances in Modeling Earth Systems, vol. 6, no. 4, pp. 1065--1094, 2014. {Online}. Available: http://dx.doi.org/10.1002/2014MS000363

Crossref

Google Scholar

[19]

D. Quinlan and C. Liao, "The ROSE Source-to-Source Compiler Infrastructure," in Cetus Users and Compiler Infrastructure Workshop, in conjunction with PACT 2011, October 2011.

Google Scholar

Cited By

View all

Mascagni MDennis JDobbins BKerr CKim Y(2019)Optimizing the HOMME dynamical core for multicore platformsInternational Journal of High Performance Computing Applications10.1177/109434201984961833:5(1030-1045)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1177/1094342019849618
Xiao JLi SWu BZhang HLi KYao EZhang YTan G(2018)Communication-Avoiding for Dynamical Core of Atmospheric General Circulation ModelProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225140(1-10)Online publication date: 13-Aug-2018
https://dl.acm.org/doi/10.1145/3225058.3225140

Recommendations

Redesigning CAM-SE for peta-scale climate modeling performance and ultra-high resolution on Sunway TaihuLight
SC '17: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

The Community Atmosphere Model (CAM) is ported, redesigned, and scaled to the full system of the Sunway TaihuLight, and provides peta-scale climate modeling performance. We refactored and optimized the complete code using OpenACC directives at the first ...
Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer

The Sunway TaihuLight supercomputer is powered by SW26010, a new 260-core processor designed with on-chip fusion of heterogeneous cores. In this article, we present our work on optimizing the training process of convolutional neural networks (CNNs) on ...
Refactoring and Optimizing WRF Model on Sunway TaihuLight
ICPP '19: Proceedings of the 48th International Conference on Parallel Processing

The Weather Research and Forecasting (WRF) Model is one of the widely-used mesoscale numerical weather prediction system and is designed for both atmospheric research and operational forecasting applications. However, it is an extremely time-consuming ...

Comments

Information & Contributors

Information

Published In

SC '16: Proceedings of the International Conference for High Performance Computing, Networking, Storage and Analysis

November 2016

1034 pages

ISBN:9781467388153

Conference Chair:
John West
University of Texas at Austin

In-Cooperation

SIGHPC: ACM Special Interest Group on High Performance Computing, Special Interest Group on High Performance Computing

Publisher

IEEE Press

Publication History

Published: 13 November 2016

Check for updates

Author Tags

Qualifiers

Research-article

Conference

SC16

Sponsor:

SIGARCH
IEEE-CS

SC16: The International Conference for High Performance Computing, Networking, Storage and Analysis

November 13 - 18, 2016

Utah, Salt Lake City

Acceptance Rates

SC '16 Paper Acceptance Rate 81 of 442 submissions, 18%;

Overall Acceptance Rate 1,516 of 6,373 submissions, 24%

Contributors

Other Metrics

View Article Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

2
Total Citations
View Citations
227
Total Downloads

Downloads (Last 12 months)6
Downloads (Last 6 weeks)1

Reflects downloads up to 27 Jan 2025

Other Metrics

View Author Metrics

Citations

Cited By

View all

Mascagni MDennis JDobbins BKerr CKim Y(2019)Optimizing the HOMME dynamical core for multicore platformsInternational Journal of High Performance Computing Applications10.1177/109434201984961833:5(1030-1045)Online publication date: 1-Sep-2019
https://dl.acm.org/doi/10.1177/1094342019849618
Xiao JLi SWu BZhang HLi KYao EZhang YTan G(2018)Communication-Avoiding for Dynamical Core of Atmospheric General Circulation ModelProceedings of the 47th International Conference on Parallel Processing10.1145/3225058.3225140(1-10)Online publication date: 13-Aug-2018
https://dl.acm.org/doi/10.1145/3225058.3225140

View Options

Login options

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Get this Publication

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Figures

Tables

Media

Affiliations

Haohuan Fu

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Junfeng Liao

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Wei Xue

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Lanning Wang

Beijing Normal University, Beijing, China

View Profile

Dexun Chen

Tsinghua University, Beijing, China

View Profile

Long Gu

National Research Center Of Parallel Computer Engineering and Technology, Beijing, China

View Profile

Jinxiu Xu

National Research Center Of Parallel Computer Engineering and Technology, Beijing, China

View Profile

Nan Ding

Tsinghua University, Beijing, China

View Profile

Xinliang Wang

Tsinghua University, Beijing, China

View Profile

Conghui He

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Shizhen Xu

Tsinghua University, Beijing, China

View Profile

Yishuang Liang

Beijing Normal University, Beijing, China

View Profile

Jiarui Fang

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Yuanchao Xu

Tsinghua University, Beijing, China

View Profile

Weijie Zheng

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Jingheng Xu

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Zhen Zheng

Tsinghua University, Beijing, China

View Profile

Wanjing Wei

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Xu Ji

Tsinghua University, Beijing, China

View Profile

He Zhang

Tsinghua University, Beijing, China

View Profile

Bingwei Chen

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Kaiwei Li

Tsinghua University, Beijing, China

View Profile

Xiaomeng Huang

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Wenguang Chen

Tsinghua University, Beijing, China

View Profile

Guangwen Yang

Tsinghua University, Beijing, China and National Supercomputing Center in Wuxi, Wuxi, China

View Profile

Cited By

Recommendations

Redesigning CAM-SE for peta-scale climate modeling performance and ultra-high resolution on Sunway TaihuLight

Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer

Refactoring and Optimizing WRF Model on Sunway TaihuLight

Comments

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Other Metrics

Article Metrics

Other Metrics

Cited By

Login options

Full Access

PDF

eReader

Abstract

References

Cited By

Recommendations

Redesigning CAM-SE for peta-scale climate modeling performance and ultra-high resolution on Sunway TaihuLight

Optimizing Convolutional Neural Networks on the Sunway TaihuLight Supercomputer

Refactoring and Optimizing WRF Model on Sunway TaihuLight

Comments

Information

Published In

Sponsors

In-Cooperation

Publisher

Publication History

Check for updates

Author Tags

Qualifiers

Conference

Acceptance Rates

Upcoming Conference

Contributors

Other Metrics

Bibliometrics

Article Metrics

Other Metrics

Citations

Cited By

Login options

Full Access

View options

PDF

eReader

Share

Share this Publication link

Share on social media

Affiliations