Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems

Peter Schwartz; Dali Wang; Fengming Yuan; Peter Thornton

doi:10.1007/978-3-031-36805-9_19

Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems

Peter Schwartz, Dali Wang, Fengming Yuan, Peter Thornton

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.

Original language	English
Title of host publication	Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings
Editors	Osvaldo Gervasi, Beniamino Murgante, David Taniar, Bernady O. Apduhan, Ana Cristina Braga, Chiara Garau, Anastasia Stratigea
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	277-290
Number of pages	14
ISBN (Print)	9783031368042
DOIs	https://doi.org/10.1007/978-3-031-36805-9_19
State	Published - 2023
Event	23rd International Conference on Computational Science and Its Applications , ICCSA 2023 - Athens, Greece Duration: Jul 3 2023 → Jul 6 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13956 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	23rd International Conference on Computational Science and Its Applications , ICCSA 2023
Country/Territory	Greece
City	Athens
Period	07/3/23 → 07/6/23

Funding

Acknowledgments. This work has been supported by FCT - Funda¸cão para a Ciência e Tecnologia within the R&D Units Project Scope: UIDB/00319/2020 and the project “QOMPASS .: Solu¸cão de Gestão de Servi¸cos de Atendimento multi-entidade, multi-servi¸co e multi-idioma” within the Project Scope NORTE-01-0247-FEDER-038462. Acknowledgments. This study was grant No. 21-71-00039. Acknowledgement. This work was supported in part by Sunway University and Sunway Business School under Kick Start Grant Scheme (KSGS) NO: GRTIN-KSGS-DBA[S]-02-2022. This work is also part of the Sustainable Business Research Cluster and Research Centre for Human-Machine Collaboration (HUMAC) at Sunway University. We also wish to thank those who have supported this research. Supported by organization Conselho Nacional de Desenvolvimento Científico e Tec-nológico (CNPq). Acknowledgements. This study was financed in part by the Coordena¸cão de Aper-fei¸coamento de Pessoal de Ńıvel Superior - Brasil (CAPES). ICCSA 2023 was organized by the National Technical University of Athens (Greece), the University of the Aegean (Greece), the University of Perugia (Italy), the University of Basilicata (Italy), Monash University (Australia), Kyushu Sangyo University (Japan), the University of Minho (Portugal). The conference was supported by two NTUA Schools, namely the School of Rural, Surveying and Geoinformatics Engineering and the School of Electrical and Computer Engineering. This study was nanced in part by the Coordenac ao de Aperfeicoamento de Pessoal de Nvel Superior - Brasil (CAPES) - Finance Code 001. This work was partially funded by CNPq, CAPES, FINEP and Fapemig. Acknowledgement. This research was supported by the MUNI Award in Science and Humanities (MASH Belarus) of the Grant Agency of Masaryk University under the Digital City project (MUNI/J/0008/2021). The work of Stanislav Sobolevsky was also partially supported by ERDF “CyberSecurity, CyberCrime and Critical Information Infrastructures Center of Excellence” (No. CZ.02.1.01/0.0/0.0/16 019/0000822). Acknowledgements. The author sincerely acknowledges the financial support under Supported by organization LNCC/SEPAD. University of Perugia, Italy Universidade Nova de Lisboa, Portugal University of Beira Interior, Portugal University of Almeria, Spain University of Salerno, Italy Erciyes University, Turkey University of Naples “Federico II”, Italy Sungkyunkwan University, Korea Sunway University, Malaysia Sungkyunkwan University, Korea Polytechnic Institute of Viana do Castelo, Portugal Federal University of Bahia, Brazil INFN, Italy Universidade Federal do Rio Grande do Sul, Brazil The Council for Scientific and Industrial Research (CSIR), South Africa Instituto Tecnológico de Informática, Spain Kausan University of Technology, Lithuania London South Bank University, UK Memorial University of Newfoundland, Canada University of Coimbra, Portugal University of L’Aquila, Italy NetApp, India/USA University of Perugia, Italy University of Minho, Portugal U.S. DOE Ames Laboratory, USA Polytechnic Institute of Bragança, Portugal National Centre for Biotechnology, CSIS, Spain Polytechnic Institute of Bragança, Portugal University of Aveiro, Portugal by the Russian Science Foundation Acknowledgements. The authors would like to thank Conselho Nacional de Desen-volvimento Científico e Tecnológico (CNPq), Coordenação de Aperfeiçoamento de Pes-soal de Nível Superior (Capes) and Pontifícia Universidade Católica do Rio de Janeiro (PUC-Rio) for their financial support. Supported by the Government of Spain, Department of Science, Innovation and Universities; European Commision project: PID2021-125871NB-I00.

Keywords

Compiler Directives
E3SM Land Model
Exascale Energy Earth System Model
OpenACC
Ultrahigh-Resolution ELM

Access to Document

10.1007/978-3-031-36805-9_19

Cite this

Schwartz, P., Wang, D., Yuan, F., & Thornton, P. (2023). Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems. In O. Gervasi, B. Murgante, D. Taniar, B. O. Apduhan, A. C. Braga, C. Garau, & A. Stratigea (Eds.), Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings (pp. 277-290). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13956 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-36805-9_19

Schwartz, Peter ; Wang, Dali ; Yuan, Fengming et al. / Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems. Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings. editor / Osvaldo Gervasi ; Beniamino Murgante ; David Taniar ; Bernady O. Apduhan ; Ana Cristina Braga ; Chiara Garau ; Anastasia Stratigea. Springer Science and Business Media Deutschland GmbH, 2023. pp. 277-290 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{bcaaa33773e046679ab35b6236e6f571,

title = "Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems",

abstract = "Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.",

keywords = "Compiler Directives, E3SM Land Model, Exascale Energy Earth System Model, OpenACC, Ultrahigh-Resolution ELM",

author = "Peter Schwartz and Dali Wang and Fengming Yuan and Peter Thornton",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 23rd International Conference on Computational Science and Its Applications , ICCSA 2023 ; Conference date: 03-07-2023 Through 06-07-2023",

year = "2023",

doi = "10.1007/978-3-031-36805-9_19",

language = "English",

isbn = "9783031368042",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "277--290",

editor = "Osvaldo Gervasi and Beniamino Murgante and David Taniar and Apduhan, {Bernady O.} and Braga, {Ana Cristina} and Chiara Garau and Anastasia Stratigea",

booktitle = "Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings",

}

Schwartz, P, Wang, D, Yuan, F & Thornton, P 2023, Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems. in O Gervasi, B Murgante, D Taniar, BO Apduhan, AC Braga, C Garau & A Stratigea (eds), Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13956 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 277-290, 23rd International Conference on Computational Science and Its Applications , ICCSA 2023, Athens, Greece, 07/3/23. https://doi.org/10.1007/978-3-031-36805-9_19

Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems. / Schwartz, Peter; Wang, Dali; Yuan, Fengming et al.
Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings. ed. / Osvaldo Gervasi; Beniamino Murgante; David Taniar; Bernady O. Apduhan; Ana Cristina Braga; Chiara Garau; Anastasia Stratigea. Springer Science and Business Media Deutschland GmbH, 2023. p. 277-290 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13956 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems

AU - Schwartz, Peter

AU - Wang, Dali

AU - Yuan, Fengming

AU - Thornton, Peter

PY - 2023

Y1 - 2023

N2 - Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.

AB - Designing and refactoring complex scientific code, such as the E3SM land model (ELM), for new computing architectures is challenging. This paper presents design strategies and technical approaches to develop a data-oriented, GPU-ready ELM model using compiler directives (OpenACC/OpenMP). We first analyze the datatypes and processes in the original ELM code. Then we present design considerations for ultrahigh-resolution ELM (uELM) development for massive GPU systems. These techniques include the global data-oriented simulation workflow, domain partition, code porting and data copy, memory reduction, parallel loop restructure and flattening, and race condition detection. We implemented the first version of uELM using OpenACC targeting the NVidia GPUs in the Summit supercomputer at Oak Ridge National Laboratory. During the implementation, we developed a software tool (named SPEL) to facilitate code generation, verification, and performance tuning using these techniques. The first uELM implementation for Nvidia GPUs on Summit delivered promising results: 1) over 98% of the ELM code was automatically generated and tuned by scripts. Most ELM modules had better computational performances than the original ELM code for CPUs. The GPU-ready uELM is more scalable than the CPU code on fully-loaded Summit nodes. Example profiling results from several modules are also presented to illustrate the performance improvements and race condition detection. The lessons learned and toolkit developed in the study are also suitable for further uELM deployment using OpenMP on the first US exascale computer, Frontier, equipped with AMD CPUs and GPUs.

KW - Compiler Directives

KW - E3SM Land Model

KW - Exascale Energy Earth System Model

KW - OpenACC

KW - Ultrahigh-Resolution ELM

UR - http://www.scopus.com/inward/record.url?scp=85164923291&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-36805-9_19

DO - 10.1007/978-3-031-36805-9_19

M3 - Conference contribution

AN - SCOPUS:85164923291

SN - 9783031368042

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 277

EP - 290

BT - Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings

A2 - Gervasi, Osvaldo

A2 - Murgante, Beniamino

A2 - Taniar, David

A2 - Apduhan, Bernady O.

A2 - Braga, Ana Cristina

A2 - Garau, Chiara

A2 - Stratigea, Anastasia

PB - Springer Science and Business Media Deutschland GmbH

T2 - 23rd International Conference on Computational Science and Its Applications , ICCSA 2023

Y2 - 3 July 2023 through 6 July 2023

ER -

Schwartz P, Wang D, Yuan F , Thornton P. Developing Ultrahigh-Resolution E3SM Land Model for GPU Systems. In Gervasi O, Murgante B, Taniar D, Apduhan BO, Braga AC, Garau C, Stratigea A, editors, Computational Science and Its Applications – ICCSA 2023 - 23rd International Conference, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 277-290. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-36805-9_19