Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI

James Buford White

doi:10.1145/3624062.3624183

Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI

James Buford White

Algorithms and Performance Analysis

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

To better advise HPC application developers, we have implemented Faces, a nearest-neighbor microbenchmark that quantifies performance trade-offs. The Faces experiments presented here explore the following design choices: 1) fewer dependent messages versus more independent messages, 2) fewer fused GPU kernels versus more simple kernels, 3) number of GPU streams, 4) size of GPU thread blocks, and 5) linear versus blocked ordering of MPI ranks. We present weak-scaling performance of a latency-sensitive "small"per-rank domain and of a bandwidth-sensitive "large"per-rank domain, and we compare results for two high-performance computers with contrasting CPU, GPU, and interconnect architectures: Summit and Frontier. We find that using more independent messages tends to give better performance than using few dependent messages. We identify performance-portability recommendations for GPU streams and synchronization, but other aspects of performance show complicated dependence on problem size and computer.

Original language	English
Title of host publication	Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Publisher	Association for Computing Machinery
Pages	1070-1080
Number of pages	11
ISBN (Electronic)	9798400707858
DOIs	https://doi.org/10.1145/3624062.3624183
State	Published - Nov 12 2023
Event	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 - Denver, United States Duration: Nov 12 2023 → Nov 17 2023

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Country/Territory	United States
City	Denver
Period	11/12/23 → 11/17/23

Funding

This research used resources of the Oak Ridge Leadership Computing Facility (OLCF) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. The OLCF Frontier Center of Excellence supported this research.

Keywords

Design Trade-Offs
GPU Kernels
GPU Streams
GPU Thread Blocks
GPU-Aware MPI
GPUs
Kernel Fusion
MPI
Nearest-Neighbor Communication
Overlap
Performance Portability
Pipelining
Programming

Access to Document

10.1145/3624062.3624183

Cite this

White, J. B. (2023). Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 (pp. 1070-1080). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3624062.3624183

White, James Buford. / Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI. Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. pp. 1070-1080 (ACM International Conference Proceeding Series).

@inproceedings{a9920741a52a49b0a1feb45df2c0f31e,

title = "Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI",

abstract = "To better advise HPC application developers, we have implemented Faces, a nearest-neighbor microbenchmark that quantifies performance trade-offs. The Faces experiments presented here explore the following design choices: 1) fewer dependent messages versus more independent messages, 2) fewer fused GPU kernels versus more simple kernels, 3) number of GPU streams, 4) size of GPU thread blocks, and 5) linear versus blocked ordering of MPI ranks. We present weak-scaling performance of a latency-sensitive {"}small{"}per-rank domain and of a bandwidth-sensitive {"}large{"}per-rank domain, and we compare results for two high-performance computers with contrasting CPU, GPU, and interconnect architectures: Summit and Frontier. We find that using more independent messages tends to give better performance than using few dependent messages. We identify performance-portability recommendations for GPU streams and synchronization, but other aspects of performance show complicated dependence on problem size and computer.",

keywords = "Design Trade-Offs, GPU Kernels, GPU Streams, GPU Thread Blocks, GPU-Aware MPI, GPUs, Kernel Fusion, MPI, Nearest-Neighbor Communication, Overlap, Performance Portability, Pipelining, Programming",

author = "White, {James Buford}",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 ; Conference date: 12-11-2023 Through 17-11-2023",

year = "2023",

month = nov,

day = "12",

doi = "10.1145/3624062.3624183",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "1070--1080",

booktitle = "Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023",

}

White, JB 2023, Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI. in Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 1070-1080, 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023, Denver, United States, 11/12/23. https://doi.org/10.1145/3624062.3624183

Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI. / White, James Buford.
Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. p. 1070-1080 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI

AU - White, James Buford

PY - 2023/11/12

Y1 - 2023/11/12

N2 - To better advise HPC application developers, we have implemented Faces, a nearest-neighbor microbenchmark that quantifies performance trade-offs. The Faces experiments presented here explore the following design choices: 1) fewer dependent messages versus more independent messages, 2) fewer fused GPU kernels versus more simple kernels, 3) number of GPU streams, 4) size of GPU thread blocks, and 5) linear versus blocked ordering of MPI ranks. We present weak-scaling performance of a latency-sensitive "small"per-rank domain and of a bandwidth-sensitive "large"per-rank domain, and we compare results for two high-performance computers with contrasting CPU, GPU, and interconnect architectures: Summit and Frontier. We find that using more independent messages tends to give better performance than using few dependent messages. We identify performance-portability recommendations for GPU streams and synchronization, but other aspects of performance show complicated dependence on problem size and computer.

AB - To better advise HPC application developers, we have implemented Faces, a nearest-neighbor microbenchmark that quantifies performance trade-offs. The Faces experiments presented here explore the following design choices: 1) fewer dependent messages versus more independent messages, 2) fewer fused GPU kernels versus more simple kernels, 3) number of GPU streams, 4) size of GPU thread blocks, and 5) linear versus blocked ordering of MPI ranks. We present weak-scaling performance of a latency-sensitive "small"per-rank domain and of a bandwidth-sensitive "large"per-rank domain, and we compare results for two high-performance computers with contrasting CPU, GPU, and interconnect architectures: Summit and Frontier. We find that using more independent messages tends to give better performance than using few dependent messages. We identify performance-portability recommendations for GPU streams and synchronization, but other aspects of performance show complicated dependence on problem size and computer.

KW - Design Trade-Offs

KW - GPU Kernels

KW - GPU Streams

KW - GPU Thread Blocks

KW - GPU-Aware MPI

KW - GPUs

KW - Kernel Fusion

KW - MPI

KW - Nearest-Neighbor Communication

KW - Overlap

KW - Performance Portability

KW - Pipelining

KW - Programming

UR - http://www.scopus.com/inward/record.url?scp=85178094826&partnerID=8YFLogxK

U2 - 10.1145/3624062.3624183

DO - 10.1145/3624062.3624183

M3 - Conference contribution

AN - SCOPUS:85178094826

T3 - ACM International Conference Proceeding Series

SP - 1070

EP - 1080

BT - Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

PB - Association for Computing Machinery

T2 - 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

Y2 - 12 November 2023 through 17 November 2023

ER -

White JB. Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery. 2023. p. 1070-1080. (ACM International Conference Proceeding Series). doi: 10.1145/3624062.3624183

Performance Portability of Programming Strategies for Nearest-Neighbor Communication with GPU-Aware MPI

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this