Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer

Matthew R. Norman; David A. Bader; Christopher Eldred; Walter M. Hannah; Benjamin R. Hillman; Christopher R. Jones; Jungmin M. Lee; L. R. Leung; Isaac Lyngaas; Kyle G. Pressel; Sarat Sreepathi; Mark A. Taylor; Xingqiu Yuan

doi:10.1177/10943420211027539

Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer

Matthew R. Norman, David A. Bader, Christopher Eldred, Walter M. Hannah, Benjamin R. Hillman, Christopher R. Jones, Jungmin M. Lee, L. R. Leung, Isaac Lyngaas, Kyle G. Pressel, Sarat Sreepathi, Mark A. Taylor, Xingqiu Yuan

Research output: Contribution to journal › Article › peer-review

15 Scopus citations

Abstract

Clouds represent a key uncertainty in future climate projection. While explicit cloud resolution remains beyond our computational grasp for global climate, we can incorporate important cloud effects through a computational middle ground called the Multi-scale Modeling Framework (MMF), also known as Super Parameterization. This algorithmic approach embeds high-resolution Cloud Resolving Models (CRMs) to represent moist convective processes within each grid column in a Global Climate Model (GCM). The MMF code requires no parallel data transfers and provides a self-contained target for acceleration. This study investigates the performance of the Energy Exascale Earth System Model-MMF (E3SM-MMF) code on the OLCF Summit supercomputer at an unprecedented scale of simulation. Hundreds of kernels in the roughly 10K lines of code in the E3SM-MMF CRM were ported to GPUs with OpenACC directives. A high-resolution benchmark using 4600 nodes on Summit demonstrates the computational capability of the GPU-enabled E3SM-MMF code in a full physics climate simulation.

Original language	English
Pages (from-to)	93-105
Number of pages	13
Journal	International Journal of High Performance Computing Applications
Volume	36
Issue number	1
DOIs	https://doi.org/10.1177/10943420211027539
State	Published - Jan 2022

Bibliographical note

Publisher Copyright:
© The Author(s) 2021.

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the US Department of Energy Office of Science and the National Nuclear Security Administration.This research was supported as part of the Energy Exascale Earth System Model (E3SM) project, funded by the US Department of Energy, Office of Science, Office of Biological and Environmental Research.This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. This work was performed under the auspices of the US Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344.The E3SM code can be accessed publicly at https://github.com/E3SM-Project/E3SM , and the CRM code can be accessed here: https://github.com/E3SM-Project/E3SM/tree/2b7c30b64c206e263b885300008488e62370f4db/components/eam/src/physics/crm/sam . Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the US Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the US Department of Energy or the United States Government.

Funders	Funder number
U.S. Department of Energy
Office of Science	DE-AC05-00OR22725
National Nuclear Security Administration	DE-NA0003525
Lawrence Livermore National Laboratory	DE-AC52-07NA27344

Keywords

GPU
OpenACC
climate
clouds

Access to Document

10.1177/10943420211027539

Cite this

Norman, M. R., Bader, D. A., Eldred, C., Hannah, W. M., Hillman, B. R., Jones, C. R., Lee, J. M., Leung, L. R., Lyngaas, I., Pressel, K. G., Sreepathi, S., Taylor, M. A., & Yuan, X. (2022). Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer. International Journal of High Performance Computing Applications, 36(1), 93-105. https://doi.org/10.1177/10943420211027539

@article{6d56724bbc144948935aedcb11ec7039,

title = "Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF{\textquoteright}s summit supercomputer",

abstract = "Clouds represent a key uncertainty in future climate projection. While explicit cloud resolution remains beyond our computational grasp for global climate, we can incorporate important cloud effects through a computational middle ground called the Multi-scale Modeling Framework (MMF), also known as Super Parameterization. This algorithmic approach embeds high-resolution Cloud Resolving Models (CRMs) to represent moist convective processes within each grid column in a Global Climate Model (GCM). The MMF code requires no parallel data transfers and provides a self-contained target for acceleration. This study investigates the performance of the Energy Exascale Earth System Model-MMF (E3SM-MMF) code on the OLCF Summit supercomputer at an unprecedented scale of simulation. Hundreds of kernels in the roughly 10K lines of code in the E3SM-MMF CRM were ported to GPUs with OpenACC directives. A high-resolution benchmark using 4600 nodes on Summit demonstrates the computational capability of the GPU-enabled E3SM-MMF code in a full physics climate simulation.",

keywords = "GPU, OpenACC, climate, clouds",

author = "Norman, {Matthew R.} and Bader, {David A.} and Christopher Eldred and Hannah, {Walter M.} and Hillman, {Benjamin R.} and Jones, {Christopher R.} and Lee, {Jungmin M.} and Leung, {L. R.} and Isaac Lyngaas and Pressel, {Kyle G.} and Sarat Sreepathi and Taylor, {Mark A.} and Xingqiu Yuan",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2021.",

year = "2022",

month = jan,

doi = "10.1177/10943420211027539",

language = "English",

volume = "36",

pages = "93--105",

journal = "International Journal of High Performance Computing Applications",

issn = "1094-3420",

publisher = "SAGE Publications",

number = "1",

}

Norman, MR, Bader, DA, Eldred, C, Hannah, WM, Hillman, BR, Jones, CR, Lee, JM, Leung, LR, Lyngaas, I, Pressel, KG, Sreepathi, S, Taylor, MA & Yuan, X 2022, 'Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer', International Journal of High Performance Computing Applications, vol. 36, no. 1, pp. 93-105. https://doi.org/10.1177/10943420211027539

TY - JOUR

T1 - Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer

AU - Norman, Matthew R.

AU - Bader, David A.

AU - Eldred, Christopher

AU - Hannah, Walter M.

AU - Hillman, Benjamin R.

AU - Jones, Christopher R.

AU - Lee, Jungmin M.

AU - Leung, L. R.

AU - Lyngaas, Isaac

AU - Pressel, Kyle G.

AU - Sreepathi, Sarat

AU - Taylor, Mark A.

AU - Yuan, Xingqiu

PY - 2022/1

Y1 - 2022/1

N2 - Clouds represent a key uncertainty in future climate projection. While explicit cloud resolution remains beyond our computational grasp for global climate, we can incorporate important cloud effects through a computational middle ground called the Multi-scale Modeling Framework (MMF), also known as Super Parameterization. This algorithmic approach embeds high-resolution Cloud Resolving Models (CRMs) to represent moist convective processes within each grid column in a Global Climate Model (GCM). The MMF code requires no parallel data transfers and provides a self-contained target for acceleration. This study investigates the performance of the Energy Exascale Earth System Model-MMF (E3SM-MMF) code on the OLCF Summit supercomputer at an unprecedented scale of simulation. Hundreds of kernels in the roughly 10K lines of code in the E3SM-MMF CRM were ported to GPUs with OpenACC directives. A high-resolution benchmark using 4600 nodes on Summit demonstrates the computational capability of the GPU-enabled E3SM-MMF code in a full physics climate simulation.

AB - Clouds represent a key uncertainty in future climate projection. While explicit cloud resolution remains beyond our computational grasp for global climate, we can incorporate important cloud effects through a computational middle ground called the Multi-scale Modeling Framework (MMF), also known as Super Parameterization. This algorithmic approach embeds high-resolution Cloud Resolving Models (CRMs) to represent moist convective processes within each grid column in a Global Climate Model (GCM). The MMF code requires no parallel data transfers and provides a self-contained target for acceleration. This study investigates the performance of the Energy Exascale Earth System Model-MMF (E3SM-MMF) code on the OLCF Summit supercomputer at an unprecedented scale of simulation. Hundreds of kernels in the roughly 10K lines of code in the E3SM-MMF CRM were ported to GPUs with OpenACC directives. A high-resolution benchmark using 4600 nodes on Summit demonstrates the computational capability of the GPU-enabled E3SM-MMF code in a full physics climate simulation.

KW - GPU

KW - OpenACC

KW - climate

KW - clouds

UR - http://www.scopus.com/inward/record.url?scp=85110837416&partnerID=8YFLogxK

U2 - 10.1177/10943420211027539

DO - 10.1177/10943420211027539

M3 - Article

AN - SCOPUS:85110837416

SN - 1094-3420

VL - 36

SP - 93

EP - 105

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 1

ER -

Unprecedented cloud resolution in a GPU-enabled full-physics atmospheric climate simulation on OLCF’s summit supercomputer

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this