Scalable Small Message Aggregation on Modern Interconnects

Aaron Welch; Oscar Hernandez; Stephen Poole; Wendy Poole

doi:10.1007/978-3-031-97492-2_11

Scalable Small Message Aggregation on Modern Interconnects

Aaron Welch
, Oscar Hernandez
, Stephen Poole
, Wendy Poole

Intelligent Systems and Facilities

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

Abstract

The partitioned global address space (PGAS) model is popular for applying a classic shared memory approach to large systems, but some classes of problems rely on large numbers of small remote memory accesses targeting random locations across the network. On modern interconnects this can overwhelm the network, leading to message rate inefficiencies. This small message problem can be solved through aggregation strategies, however these typically require undesirable code restructuring that is cumbersome to incorporate and maintain in user applications. A strategy called “aggregation contexts” aimed at alleviating this burden has previously been proposed for the OpenSHMEM PGAS API. Despite its potential, it has not yet been validated for scalability on large systems consisting of thousands of nodes, nor proven to be performance-portable, which are critical for its adoption. In this paper, we demonstrate the scalability and performance portability of aggregation contexts using up to 8192 nodes on ORNL’s Frontier system. Our study reveals good scaling patterns while also identifying further opportunities for performance improvements to make it even more effective.

Original language	English
Title of host publication	Lecture Notes in Computer Science
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	103-113
Number of pages	11
DOIs	https://doi.org/10.1007/978-3-031-97492-2_11
State	Published - 2025

Publication series

Name	Lecture Notes in Computer Science
Volume	LNCS 14564
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research used the Frontier and Andes resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. This work was funded through the Strategic Partnership Projects Funding Office via Los Alamos National Laboratory with IAN 61921590 for the project. Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy. gov/downloads/doe-public-access-plan). This research used the Frontier and Andes resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. This work was funded through the Strategic Partnership Projects Funding Office via Los Alamos National Laboratory with IAN 61921590 for the project.

Keywords

OpenSHMEM
aggregation contexts
conveyors
many-to-many communication patterns
message aggregation

Access to Document

10.1007/978-3-031-97492-2_11

Cite this

@inbook{738f917197ab4aaba229c4013637d067,

title = "Scalable Small Message Aggregation on Modern Interconnects",

abstract = "The partitioned global address space (PGAS) model is popular for applying a classic shared memory approach to large systems, but some classes of problems rely on large numbers of small remote memory accesses targeting random locations across the network. On modern interconnects this can overwhelm the network, leading to message rate inefficiencies. This small message problem can be solved through aggregation strategies, however these typically require undesirable code restructuring that is cumbersome to incorporate and maintain in user applications. A strategy called “aggregation contexts” aimed at alleviating this burden has previously been proposed for the OpenSHMEM PGAS API. Despite its potential, it has not yet been validated for scalability on large systems consisting of thousands of nodes, nor proven to be performance-portable, which are critical for its adoption. In this paper, we demonstrate the scalability and performance portability of aggregation contexts using up to 8192 nodes on ORNL{\textquoteright}s Frontier system. Our study reveals good scaling patterns while also identifying further opportunities for performance improvements to make it even more effective.",

keywords = "OpenSHMEM, aggregation contexts, conveyors, many-to-many communication patterns, message aggregation",

author = "Aaron Welch and Oscar Hernandez and Stephen Poole and Wendy Poole",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.",

year = "2025",

doi = "10.1007/978-3-031-97492-2\_11",

language = "English",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "103--113",

booktitle = "Lecture Notes in Computer Science",

}

TY - CHAP

T1 - Scalable Small Message Aggregation on Modern Interconnects

AU - Welch, Aaron

AU - Hernandez, Oscar

AU - Poole, Stephen

AU - Poole, Wendy

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

PY - 2025

Y1 - 2025

N2 - The partitioned global address space (PGAS) model is popular for applying a classic shared memory approach to large systems, but some classes of problems rely on large numbers of small remote memory accesses targeting random locations across the network. On modern interconnects this can overwhelm the network, leading to message rate inefficiencies. This small message problem can be solved through aggregation strategies, however these typically require undesirable code restructuring that is cumbersome to incorporate and maintain in user applications. A strategy called “aggregation contexts” aimed at alleviating this burden has previously been proposed for the OpenSHMEM PGAS API. Despite its potential, it has not yet been validated for scalability on large systems consisting of thousands of nodes, nor proven to be performance-portable, which are critical for its adoption. In this paper, we demonstrate the scalability and performance portability of aggregation contexts using up to 8192 nodes on ORNL’s Frontier system. Our study reveals good scaling patterns while also identifying further opportunities for performance improvements to make it even more effective.

AB - The partitioned global address space (PGAS) model is popular for applying a classic shared memory approach to large systems, but some classes of problems rely on large numbers of small remote memory accesses targeting random locations across the network. On modern interconnects this can overwhelm the network, leading to message rate inefficiencies. This small message problem can be solved through aggregation strategies, however these typically require undesirable code restructuring that is cumbersome to incorporate and maintain in user applications. A strategy called “aggregation contexts” aimed at alleviating this burden has previously been proposed for the OpenSHMEM PGAS API. Despite its potential, it has not yet been validated for scalability on large systems consisting of thousands of nodes, nor proven to be performance-portable, which are critical for its adoption. In this paper, we demonstrate the scalability and performance portability of aggregation contexts using up to 8192 nodes on ORNL’s Frontier system. Our study reveals good scaling patterns while also identifying further opportunities for performance improvements to make it even more effective.

KW - OpenSHMEM

KW - aggregation contexts

KW - conveyors

KW - many-to-many communication patterns

KW - message aggregation

UR - https://www.scopus.com/pages/publications/105010317634

U2 - 10.1007/978-3-031-97492-2_11

DO - 10.1007/978-3-031-97492-2_11

M3 - Chapter

AN - SCOPUS:105010317634

T3 - Lecture Notes in Computer Science

SP - 103

EP - 113

BT - Lecture Notes in Computer Science

PB - Springer Science and Business Media Deutschland GmbH

ER -