Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA

Bo Zhang; Philip E. Davis; Nicolas Morales; Zhao Zhang; Keita Teranishi; Manish Parashar

doi:10.1007/978-3-031-39698-4_22

Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA

Bo Zhang
, Philip E. Davis
, Nicolas Morales
, Zhao Zhang
, Keita Teranishi
, Manish Parashar

Programming Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53% and 40% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.

Original language	English
Title of host publication	Euro-Par 2023
Subtitle of host publication	Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings
Editors	José Cano, Marios D. Dikaiakos, George A. Papadopoulos, Miquel Pericàs, Rizos Sakellariou
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	323-338
Number of pages	16
ISBN (Print)	9783031396977
DOIs	https://doi.org/10.1007/978-3-031-39698-4_22
State	Published - 2023
Event	29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023 - Limassol, Cyprus Duration: Aug 28 2023 → Sep 1 2023

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14100 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023
Country/Territory	Cyprus
City	Limassol
Period	08/28/23 → 09/1/23

Funding

National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (NNSA) under contract DE-NA0003525. This work was funded by NNSA’s Advanced Simulation and Computing (ASC) Program. This manuscript has been authored by UT-Battelle LLC under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). This work is also based upon work by the RAPIDS2 Institute supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research through the Advanced Computing (SciDAC) program under Award Number DE-SC0023130. The datasets and code generated during and/or analysed during the current study are available in the Figshare repository: https://doi.org/10.6084/m9.figshare.23535855 [4]. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government. This research used resources of the National Energy Research Scientific Computing Center, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration (NNSA) under contract DE-NA0003525. This work was funded by NNSA’s Advanced Simulation and Computing (ASC) Program. This manuscript has been authored by UT-Battelle LLC under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). This work is also based upon work by the RAPIDS2 Institute supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research through the Advanced Computing (SciDAC) program under Award Number DE-SC0023130. The datasets and code generated during and/or analysed during the current study are available in the Figshare repository: https://doi.org/10.6084/m9.figshare.23535855 [4]. This paper describes objective technical results and analysis. Any subjective views or opinions that might be expressed in the paper do not necessarily represent the views of the U.S. Department of Energy or the United States Government.

Keywords

Extreme-Scale Data Management
GPU
GPUDirect RDMA
In-Situ
Workflow

Access to Document

10.1007/978-3-031-39698-4_22

Cite this

Zhang, B., Davis, P. E., Morales, N., Zhang, Z., Teranishi, K., & Parashar, M. (2023). Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA. In J. Cano, M. D. Dikaiakos, G. A. Papadopoulos, M. Pericàs, & R. Sakellariou (Eds.), Euro-Par 2023: Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings (pp. 323-338). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14100 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-39698-4_22

Zhang, Bo ; Davis, Philip E. ; Morales, Nicolas et al. / Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA. Euro-Par 2023: Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings. editor / José Cano ; Marios D. Dikaiakos ; George A. Papadopoulos ; Miquel Pericàs ; Rizos Sakellariou. Springer Science and Business Media Deutschland GmbH, 2023. pp. 323-338 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{d7e33c8fde3f44759e53ad95b9344216,

title = "Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA",

abstract = "The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53\% and 40\% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.",

keywords = "Extreme-Scale Data Management, GPU, GPUDirect RDMA, In-Situ, Workflow",

author = "Bo Zhang and Davis, \{Philip E.\} and Nicolas Morales and Zhao Zhang and Keita Teranishi and Manish Parashar",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023 ; Conference date: 28-08-2023 Through 01-09-2023",

year = "2023",

doi = "10.1007/978-3-031-39698-4\_22",

language = "English",

isbn = "9783031396977",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "323--338",

editor = "Jos{\'e} Cano and Dikaiakos, \{Marios D.\} and Papadopoulos, \{George A.\} and Miquel Peric{\`a}s and Rizos Sakellariou",

booktitle = "Euro-Par 2023",

}

Zhang, B, Davis, PE, Morales, N, Zhang, Z, Teranishi, K & Parashar, M 2023, Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA. in J Cano, MD Dikaiakos, GA Papadopoulos, M Pericàs & R Sakellariou (eds), Euro-Par 2023: Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14100 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 323-338, 29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023, Limassol, Cyprus, 08/28/23. https://doi.org/10.1007/978-3-031-39698-4_22

Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA. / Zhang, Bo; Davis, Philip E.; Morales, Nicolas et al.
Euro-Par 2023: Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings. ed. / José Cano; Marios D. Dikaiakos; George A. Papadopoulos; Miquel Pericàs; Rizos Sakellariou. Springer Science and Business Media Deutschland GmbH, 2023. p. 323-338 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14100 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA

AU - Zhang, Bo

AU - Davis, Philip E.

AU - Morales, Nicolas

AU - Zhang, Zhao

AU - Teranishi, Keita

AU - Parashar, Manish

PY - 2023

Y1 - 2023

N2 - The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53% and 40% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.

AB - The extreme-scale computing landscape is increasingly dominated by GPU-accelerated systems. At the same time, in-situ workflows that employ memory-to-memory inter-application data exchanges have emerged as an effective approach for leveraging these extreme-scale systems. In the case of GPUs, GPUDirect RDMA enables third-party devices, such as network interface cards, to access GPU memory directly and has been adopted for intra-application communications across GPUs. In this paper, we present an interoperable framework for GPU-based in-situ workflows that optimizes data movement using GPUDirect RDMA. Specifically, we analyze the characteristics of the possible data movement pathways between GPUs from an in-situ workflow perspective, and design a strategy that maximizes throughput. Furthermore, we implement this approach as an extension of the DataSpaces data staging service, and experimentally evaluate its performance and scalability on a current leadership GPU cluster. The performance results show that the proposed design reduces data-movement time by up to 53% and 40% for the sender and receiver, respectively, and maintains excellent scalability for up to 256 GPUs.

KW - Extreme-Scale Data Management

KW - GPU

KW - GPUDirect RDMA

KW - In-Situ

KW - Workflow

UR - https://www.scopus.com/pages/publications/85171577075

U2 - 10.1007/978-3-031-39698-4_22

DO - 10.1007/978-3-031-39698-4_22

M3 - Conference contribution

AN - SCOPUS:85171577075

SN - 9783031396977

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 323

EP - 338

BT - Euro-Par 2023

A2 - Cano, José

A2 - Dikaiakos, Marios D.

A2 - Papadopoulos, George A.

A2 - Pericàs, Miquel

A2 - Sakellariou, Rizos

PB - Springer Science and Business Media Deutschland GmbH

T2 - 29th International European Conference on Parallel and Distributed Computing, Euro-Par 2023

Y2 - 28 August 2023 through 1 September 2023

ER -

Zhang B, Davis PE, Morales N, Zhang Z, Teranishi K, Parashar M. Optimizing Data Movement for GPU-Based In-Situ Workflow Using GPUDirect RDMA. In Cano J, Dikaiakos MD, Papadopoulos GA, Pericàs M, Sakellariou R, editors, Euro-Par 2023: Parallel Processing - 29th International Conference on Parallel and Distributed Computing, Proceedings. Springer Science and Business Media Deutschland GmbH. 2023. p. 323-338. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-39698-4_22