Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications

William F. Godoy; Oscar Hernandez; Paul R.C. Kent; Maria Patrou; Kazi Asifuzzaman; Narasinga Rao Miniskar; Pedro Valero-Lara; Jeffrey S. Vetter; Matthew D. Sinclair; Jason Lowe-Power; Bobby R. Bruce

doi:10.1007/978-3-032-07612-0_14

Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We characterize the GPU energy usage of two widely adopted exascale-ready applications representing two classes of particle and mesh solvers: (i) QMCPACK, a quantum Monte Carlo package, and (ii) AMReX-Castro, an adaptive mesh astrophysical code. We analyze power, temperature, utilization, and energy traces from double-/single (mixed)-precision benchmarks on NVIDIA’s A100 and H100 and AMD’s MI250X GPUs using queries in NVML and rocm_smi_lib, respectively. We explore application-specific metrics to provide insights on energy vs. performance trade-offs. Our results suggest that mixed-precision energy savings range between 6–25% on QMCPACK and 45% on AMReX-Castro. Also, we found gaps in the AMD tooling used on Frontier GPUs that need to be understood, while query resolutions on NVML have little variability between 1 ms-1 s. Overall, application level knowledge is crucial to define energy-cost/science-benefit opportunities for the codesign of future supercomputer architectures in the post-Moore era.

Original language	English
Title of host publication	High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers
Editors	Sarah Neuwirth, Arnab Kumar Paul, Tobias Weinzierl, Erin Claire Carson
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	177-190
Number of pages	14
ISBN (Print)	9783032076113
DOIs	https://doi.org/10.1007/978-3-032-07612-0_14
State	Published - 2026
Event	40th International Conference on High Performance Computing, ISC High Performance 2025 - Hamburg, Germany Duration: Jun 10 2025 → Jun 13 2025

Publication series

Name	Lecture Notes in Computer Science
Volume	16091 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	40th International Conference on High Performance Computing, ISC High Performance 2025
Country/Territory	Germany
City	Hamburg
Period	06/10/25 → 06/13/25

Funding

This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (https://www.energy.gov/doe-public-access-plan). This material is based on work supported by the DOE’s Office of Science, Office of Advanced Scientific Computing Research through EXPRESS: 2023 Exploratory Research for Extreme Scale Science. PRCK was supported by the DOE’s Office of Science, Basic Energy Sciences, Materials Sciences and Engineering Division as part of the Computational Materials Sciences Program and the Center for Predictive Simulation of Functional Materials. This research used resources of the Oak Ridge Leadership Computing Facility and the Experimental Computing Laboratory at the Oak Ridge National Laboratory, which is supported by the DOE’s Office of Science under Contract No. DE-AC05-00OR22725. WG would like to acknowledge Brandon Tran from the University of Wisconsin for the valuable discussion on NVML.

Keywords

Energy efficiency
GPU Power
HPC Applications

Access to Document

10.1007/978-3-032-07612-0_14

Cite this

Godoy, W. F., Hernandez, O., Kent, P. R. C., Patrou, M., Asifuzzaman, K., Miniskar, N. R., Valero-Lara, P., Vetter, J. S., Sinclair, M. D., Lowe-Power, J., & Bruce, B. R. (2026). Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications. In S. Neuwirth, A. K. Paul, T. Weinzierl, & E. C. Carson (Eds.), High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers (pp. 177-190). (Lecture Notes in Computer Science; Vol. 16091 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-032-07612-0_14

Godoy, William F. ; Hernandez, Oscar ; Kent, Paul R.C. et al. / Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications. High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers. editor / Sarah Neuwirth ; Arnab Kumar Paul ; Tobias Weinzierl ; Erin Claire Carson. Springer Science and Business Media Deutschland GmbH, 2026. pp. 177-190 (Lecture Notes in Computer Science).

@inproceedings{1c160be12d0247a69b41f7052cf15509,

title = "Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications",

abstract = "We characterize the GPU energy usage of two widely adopted exascale-ready applications representing two classes of particle and mesh solvers: (i) QMCPACK, a quantum Monte Carlo package, and (ii) AMReX-Castro, an adaptive mesh astrophysical code. We analyze power, temperature, utilization, and energy traces from double-/single (mixed)-precision benchmarks on NVIDIA{\textquoteright}s A100 and H100 and AMD{\textquoteright}s MI250X GPUs using queries in NVML and rocm\_smi\_lib, respectively. We explore application-specific metrics to provide insights on energy vs. performance trade-offs. Our results suggest that mixed-precision energy savings range between 6–25\% on QMCPACK and 45\% on AMReX-Castro. Also, we found gaps in the AMD tooling used on Frontier GPUs that need to be understood, while query resolutions on NVML have little variability between 1 ms-1 s. Overall, application level knowledge is crucial to define energy-cost/science-benefit opportunities for the codesign of future supercomputer architectures in the post-Moore era.",

keywords = "Energy efficiency, GPU Power, HPC Applications",

author = "Godoy, \{William F.\} and Oscar Hernandez and Kent, \{Paul R.C.\} and Maria Patrou and Kazi Asifuzzaman and Miniskar, \{Narasinga Rao\} and Pedro Valero-Lara and Vetter, \{Jeffrey S.\} and Sinclair, \{Matthew D.\} and Jason Lowe-Power and Bruce, \{Bobby R.\}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.; 40th International Conference on High Performance Computing, ISC High Performance 2025 ; Conference date: 10-06-2025 Through 13-06-2025",

year = "2026",

doi = "10.1007/978-3-032-07612-0\_14",

language = "English",

isbn = "9783032076113",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "177--190",

editor = "Sarah Neuwirth and Paul, \{Arnab Kumar\} and Tobias Weinzierl and Carson, \{Erin Claire\}",

booktitle = "High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers",

}

Godoy, WF , Hernandez, O , Kent, PRC , Patrou, M , Asifuzzaman, K , Miniskar, NR , Valero-Lara, P , Vetter, JS, Sinclair, MD, Lowe-Power, J & Bruce, BR 2026, Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications. in S Neuwirth, AK Paul, T Weinzierl & EC Carson (eds), High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science, vol. 16091 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 177-190, 40th International Conference on High Performance Computing, ISC High Performance 2025, Hamburg, Germany, 06/10/25. https://doi.org/10.1007/978-3-032-07612-0_14

Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications. / Godoy, William F.; Hernandez, Oscar ; Kent, Paul R.C. et al.
High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers. ed. / Sarah Neuwirth; Arnab Kumar Paul; Tobias Weinzierl; Erin Claire Carson. Springer Science and Business Media Deutschland GmbH, 2026. p. 177-190 (Lecture Notes in Computer Science; Vol. 16091 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications

AU - Godoy, William F.

AU - Hernandez, Oscar

AU - Kent, Paul R.C.

AU - Patrou, Maria

AU - Asifuzzaman, Kazi

AU - Miniskar, Narasinga Rao

AU - Valero-Lara, Pedro

AU - Vetter, Jeffrey S.

AU - Sinclair, Matthew D.

AU - Lowe-Power, Jason

AU - Bruce, Bobby R.

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

PY - 2026

Y1 - 2026

N2 - We characterize the GPU energy usage of two widely adopted exascale-ready applications representing two classes of particle and mesh solvers: (i) QMCPACK, a quantum Monte Carlo package, and (ii) AMReX-Castro, an adaptive mesh astrophysical code. We analyze power, temperature, utilization, and energy traces from double-/single (mixed)-precision benchmarks on NVIDIA’s A100 and H100 and AMD’s MI250X GPUs using queries in NVML and rocm_smi_lib, respectively. We explore application-specific metrics to provide insights on energy vs. performance trade-offs. Our results suggest that mixed-precision energy savings range between 6–25% on QMCPACK and 45% on AMReX-Castro. Also, we found gaps in the AMD tooling used on Frontier GPUs that need to be understood, while query resolutions on NVML have little variability between 1 ms-1 s. Overall, application level knowledge is crucial to define energy-cost/science-benefit opportunities for the codesign of future supercomputer architectures in the post-Moore era.

AB - We characterize the GPU energy usage of two widely adopted exascale-ready applications representing two classes of particle and mesh solvers: (i) QMCPACK, a quantum Monte Carlo package, and (ii) AMReX-Castro, an adaptive mesh astrophysical code. We analyze power, temperature, utilization, and energy traces from double-/single (mixed)-precision benchmarks on NVIDIA’s A100 and H100 and AMD’s MI250X GPUs using queries in NVML and rocm_smi_lib, respectively. We explore application-specific metrics to provide insights on energy vs. performance trade-offs. Our results suggest that mixed-precision energy savings range between 6–25% on QMCPACK and 45% on AMReX-Castro. Also, we found gaps in the AMD tooling used on Frontier GPUs that need to be understood, while query resolutions on NVML have little variability between 1 ms-1 s. Overall, application level knowledge is crucial to define energy-cost/science-benefit opportunities for the codesign of future supercomputer architectures in the post-Moore era.

KW - Energy efficiency

KW - GPU Power

KW - HPC Applications

UR - https://www.scopus.com/pages/publications/105023481940

U2 - 10.1007/978-3-032-07612-0_14

DO - 10.1007/978-3-032-07612-0_14

M3 - Conference contribution

AN - SCOPUS:105023481940

SN - 9783032076113

T3 - Lecture Notes in Computer Science

SP - 177

EP - 190

BT - High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers

A2 - Neuwirth, Sarah

A2 - Paul, Arnab Kumar

A2 - Weinzierl, Tobias

A2 - Carson, Erin Claire

PB - Springer Science and Business Media Deutschland GmbH

T2 - 40th International Conference on High Performance Computing, ISC High Performance 2025

Y2 - 10 June 2025 through 13 June 2025

ER -

Godoy WF , Hernandez O , Kent PRC , Patrou M , Asifuzzaman K , Miniskar NR et al. Characterizing GPU Energy Usage in Exascale-Ready Portable Science Applications. In Neuwirth S, Paul AK, Weinzierl T, Carson EC, editors, High Performance Computing - ISC High Performance 2025 International Workshops, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2026. p. 177-190. (Lecture Notes in Computer Science). doi: 10.1007/978-3-032-07612-0_14