An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Hyun Gyu Kang; Raymond S. Tuminaro; Andrey Prokopenko; Seth R. Johnson; Andrew G. Salinger; Katherine J. Evans

doi:10.1177/10943420231205601

An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Hyun Gyu Kang, Raymond S. Tuminaro, Andrey Prokopenko, Seth R. Johnson, Andrew G. Salinger, Katherine J. Evans

Research output: Contribution to journal › Article › peer-review

Abstract

We demonstrate use of a modern Fortran solver interface to manage solver algorithms for an implicit barotropic mode solver in the Model for Predictions Across Scales-Ocean (MPAS-O). ForTrilinos, a Fortran interface to Trilinos that contains a large collection of solver capabilities written in C++, has been implemented in MPAS-O to provide access to a suite of linear solver options. By virtue of the simplified wrapper and interface generator (SWIG) automation tool that generates modern Fortran interfaces to C++ code, we were able to implement the Fortran solver interface in MPAS-O using a familiar Fortran coding style while minimizing performance degradation. The ForTrilinos solver interface is written within MPAS-O’s time stepping modules as a subroutine in conjunction with MPAS-O code. Applied to an idealized ocean and a high-resolution realistic ocean test case, parallel performance of ForTrilinos solvers is examined. It is found that parallel scalability of the ForTrilinos solvers is highly dependent on the number of global synchronization points per solver iteration in each iterative solver algorithm. ForTrilinos solvers perform best compared to the Fortran hand-crafted (FHC) solver when the amount of work per processor is large enough. However, parallel scalability is better with the FHC solver and so when the work per core is modest FHC outperforms ForTrilinos. The intercomparison between the ForTrilinos and FHC solvers reveals that this performance hit in the ForTrilinos solver mostly comes from the global synchronization process, while suggesting that the matrix-vector multiplication process in the FHC solver needs to be optimized for better performance.

Original language	English
Pages (from-to)	175-191
Number of pages	17
Journal	International Journal of High Performance Computing Applications
Volume	38
Issue number	3
DOIs	https://doi.org/10.1177/10943420231205601
State	Published - May 2024

Bibliographical note

Publisher Copyright:
© The Author(s) 2023.

Funding

This research was supported as a part of the E3SM project, funded by the U.S. Department of Energy (DOE), Office of Science, Office of Biological and Environmental Research. This research was also supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. Authors greatly appreciate the comments of two anonymous reviewers. The authors also wish to thank Philip W. Jones in Los Alamos National Laboratory for constructive suggestions and useful comments. Numerical simulations were performed at the National Energy Research Scientific Computing Center (NERSC, DE-AC05-00OR22725) which is one of the primary scientific computing facilities for the Office of Science, U.S. Department of Energy. This research also used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. MPAS-Ocean, an open-source global ocean model, was obtained from the E3SM project. E3SM data are freely available through the Earth System Grid Federation (ESGF) distributed archives. See details at https://e3sm.org/data and https://mpas-dev.github.io . Trilinos is a collection of open-source software libraries, called packages, intended to be used as building blocks for the development of scientific applications. See details at https://trilinos.github.io . ForTrilinos is an open-source software library providing object-oriented Fortran interfaces to Trilinos C++ packages. See details at https://fortrilinos.readthedocs.io . The ForTrilinos-enabled MPAS-O and initial conditions for all test cases can be found in https://zenodo.org/record/7514256 except the realistic ocean simulation uploaded in https://zenodo.org/record/1252437 .

Funders	Funder number
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science
National Nuclear Security Administration
Biological and Environmental Research	17-SC-20-SC

Keywords

ForTrilinos
Ocean model
Trilinos
implicit solver
parallel performance
solver interface

Access to Document

10.1177/10943420231205601

Cite this

@article{b0d5ff205f0449fca04cdba1fe966f74,

title = "An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface",

abstract = "We demonstrate use of a modern Fortran solver interface to manage solver algorithms for an implicit barotropic mode solver in the Model for Predictions Across Scales-Ocean (MPAS-O). ForTrilinos, a Fortran interface to Trilinos that contains a large collection of solver capabilities written in C++, has been implemented in MPAS-O to provide access to a suite of linear solver options. By virtue of the simplified wrapper and interface generator (SWIG) automation tool that generates modern Fortran interfaces to C++ code, we were able to implement the Fortran solver interface in MPAS-O using a familiar Fortran coding style while minimizing performance degradation. The ForTrilinos solver interface is written within MPAS-O{\textquoteright}s time stepping modules as a subroutine in conjunction with MPAS-O code. Applied to an idealized ocean and a high-resolution realistic ocean test case, parallel performance of ForTrilinos solvers is examined. It is found that parallel scalability of the ForTrilinos solvers is highly dependent on the number of global synchronization points per solver iteration in each iterative solver algorithm. ForTrilinos solvers perform best compared to the Fortran hand-crafted (FHC) solver when the amount of work per processor is large enough. However, parallel scalability is better with the FHC solver and so when the work per core is modest FHC outperforms ForTrilinos. The intercomparison between the ForTrilinos and FHC solvers reveals that this performance hit in the ForTrilinos solver mostly comes from the global synchronization process, while suggesting that the matrix-vector multiplication process in the FHC solver needs to be optimized for better performance.",

keywords = "ForTrilinos, Ocean model, Trilinos, implicit solver, parallel performance, solver interface",

author = "Kang, {Hyun Gyu} and Tuminaro, {Raymond S.} and Andrey Prokopenko and Johnson, {Seth R.} and Salinger, {Andrew G.} and Evans, {Katherine J.}",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2023.",

year = "2024",

month = may,

doi = "10.1177/10943420231205601",

language = "English",

volume = "38",

pages = "175--191",

journal = "International Journal of High Performance Computing Applications",

issn = "1094-3420",

publisher = "SAGE Publications",

number = "3",

}

TY - JOUR

T1 - An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

AU - Kang, Hyun Gyu

AU - Tuminaro, Raymond S.

AU - Prokopenko, Andrey

AU - Johnson, Seth R.

AU - Salinger, Andrew G.

AU - Evans, Katherine J.

N1 - Publisher Copyright: © The Author(s) 2023.

PY - 2024/5

Y1 - 2024/5

N2 - We demonstrate use of a modern Fortran solver interface to manage solver algorithms for an implicit barotropic mode solver in the Model for Predictions Across Scales-Ocean (MPAS-O). ForTrilinos, a Fortran interface to Trilinos that contains a large collection of solver capabilities written in C++, has been implemented in MPAS-O to provide access to a suite of linear solver options. By virtue of the simplified wrapper and interface generator (SWIG) automation tool that generates modern Fortran interfaces to C++ code, we were able to implement the Fortran solver interface in MPAS-O using a familiar Fortran coding style while minimizing performance degradation. The ForTrilinos solver interface is written within MPAS-O’s time stepping modules as a subroutine in conjunction with MPAS-O code. Applied to an idealized ocean and a high-resolution realistic ocean test case, parallel performance of ForTrilinos solvers is examined. It is found that parallel scalability of the ForTrilinos solvers is highly dependent on the number of global synchronization points per solver iteration in each iterative solver algorithm. ForTrilinos solvers perform best compared to the Fortran hand-crafted (FHC) solver when the amount of work per processor is large enough. However, parallel scalability is better with the FHC solver and so when the work per core is modest FHC outperforms ForTrilinos. The intercomparison between the ForTrilinos and FHC solvers reveals that this performance hit in the ForTrilinos solver mostly comes from the global synchronization process, while suggesting that the matrix-vector multiplication process in the FHC solver needs to be optimized for better performance.

AB - We demonstrate use of a modern Fortran solver interface to manage solver algorithms for an implicit barotropic mode solver in the Model for Predictions Across Scales-Ocean (MPAS-O). ForTrilinos, a Fortran interface to Trilinos that contains a large collection of solver capabilities written in C++, has been implemented in MPAS-O to provide access to a suite of linear solver options. By virtue of the simplified wrapper and interface generator (SWIG) automation tool that generates modern Fortran interfaces to C++ code, we were able to implement the Fortran solver interface in MPAS-O using a familiar Fortran coding style while minimizing performance degradation. The ForTrilinos solver interface is written within MPAS-O’s time stepping modules as a subroutine in conjunction with MPAS-O code. Applied to an idealized ocean and a high-resolution realistic ocean test case, parallel performance of ForTrilinos solvers is examined. It is found that parallel scalability of the ForTrilinos solvers is highly dependent on the number of global synchronization points per solver iteration in each iterative solver algorithm. ForTrilinos solvers perform best compared to the Fortran hand-crafted (FHC) solver when the amount of work per processor is large enough. However, parallel scalability is better with the FHC solver and so when the work per core is modest FHC outperforms ForTrilinos. The intercomparison between the ForTrilinos and FHC solvers reveals that this performance hit in the ForTrilinos solver mostly comes from the global synchronization process, while suggesting that the matrix-vector multiplication process in the FHC solver needs to be optimized for better performance.

KW - ForTrilinos

KW - Ocean model

KW - Trilinos

KW - implicit solver

KW - parallel performance

KW - solver interface

UR - http://www.scopus.com/inward/record.url?scp=85177091981&partnerID=8YFLogxK

U2 - 10.1177/10943420231205601

DO - 10.1177/10943420231205601

M3 - Article

AN - SCOPUS:85177091981

SN - 1094-3420

VL - 38

SP - 175

EP - 191

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 3

ER -

An implicit barotropic mode solver for MPAS-ocean using a modern Fortran solver interface

Abstract

Bibliographical note

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this