Performance Porting the ExaStar Multi-Physics App Thornado On Heterogeneous Systems - A Fortran-OpenMP Code-Base Evaluation

The heterogeneity of HPC systems requires efficient host-to-device porting of compute kernels and high-bandwidth data communication. This capability varies from one system to another depending on system architectures and environments. New vendors such as AMD and Intel are entering the GPU field, creating a software portability challenge. Major scientific simulation code bases rely on Fortran and require portable programming models for performance porting to HPC systems with high software productivity. Even though OpenMP target offloading features support portability, most Fortran-OpenMP code bases face significant challenges. Hence, in this work, we motivated an evaluation of a) the computing capability of heterogeneous systems for Fortran-OpenMP-based multi-physics code bases, and b) the performance portability of the astrophysical supernova simulation code Flash-X on heterogeneous systems. For this study, three HPC systems were chosen: Sunspot, a test-bed system of the Intel-PVC GPU featured supercomputer Aurora and Polaris, an NVIDIA system accelerated by A100 GPU, both located at the Argonne Leadership Computing Facility (ALCF), and the AMD-MI250-based Frontier at the Oak Ridge Leadership Computing Facility (OLCF). We discuss challenges and solutions for performance porting the compute-intensive module Thornado, which can be incorporated as an external library in Flash-X to model neutrino transport. We show that the performance of test apps improved by approximately 24× using the relevant optimization strategies + compiler-and-system updates. Further, this study helped improve the intel OneAPI-OpenMP compiler by providing bug reports and reproducers internally.