RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources

Mikhail Titov; Matteo Turilli; Andre Merzky; Thomas Naughton; Wael Elwasif; Shantenu Jha

doi:10.1007/978-3-031-22698-4_5

RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources

Mikhail Titov, Matteo Turilli, Andre Merzky, Thomas Naughton, Wael Elwasif, Shantenu Jha

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Execution of heterogeneous workflows on high-performance computing (HPC) platforms present unprecedented resource management and execution coordination challenges for runtime systems. Task heterogeneity increases the complexity of resource and execution management, limiting the scalability and efficiency of workflow execution. Resource partitioning and distribution of tasks execution over portioned resources promises to address those problems but we lack an experimental evaluation of its performance at scale. This paper provides a performance evaluation of the Process Management Interface for Exascale (PMIx) and its reference implementation PRRTE on the leadership-class HPC platform Summit, when integrated into a pilot-based runtime system called RADICAL-Pilot. We partition resources across multiple PRRTE Distributed Virtual Machine (DVM) environments, responsible for launching tasks via the PMIx interface. We experimentally measure the workload execution performance in terms of task scheduling/launching rate and distribution of DVM task placement times, DVM startup and termination overheads on the Summit leadership-class HPC platform. Integrated solution with PMIx/PRRTE enables using an abstracted, standardized set of interfaces for orchestrating the launch process, dynamic process management and monitoring capabilities. It extends scaling capabilities allowing to overcome a limitation of other launching mechanisms (e.g., JSM/LSF). Explored different DVM setup configurations provide insights on DVM performance and a layout to leverage it. Our experimental results show that heterogeneous workload of 65,500 tasks on 2048 nodes, and partitioned across 32 DVMs, runs steady with resource utilization not lower than 52 %. While having less concurrently executed tasks resource utilization is able to reach up to 85 %, based on results of heterogeneous workload of 8200 tasks on 256 nodes and 2 DVMs.

Original language	English
Title of host publication	Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers
Editors	Dalibor Klusácek, Corbalán Julita, Gonzalo P. Rodrigo
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	88-107
Number of pages	20
ISBN (Print)	9783031226977
DOIs	https://doi.org/10.1007/978-3-031-22698-4_5
State	Published - 2023
Event	25th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2022, held in conjunction with the 36th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2022 - Virtual, Online Duration: Jun 3 2022 → Jun 3 2022

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	13592 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	25th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2022, held in conjunction with the 36th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2022
City	Virtual, Online
Period	06/3/22 → 06/3/22

Funding

Acknowledgments. We would like to thank other members of the PMIx community, and Ralph Castain in particular, for the excellent work that we build upon. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work is also supported by the ExaWorks project (part of the Exascale Computing Project (ECP)) under DOE Contract No. DE-SC0012704 and by the DOE HEP Center for Computational Excellence at Brookhaven National Laboratory under B&R KA2401045. We also acknowledge DOE INCITE awards for allocations on Summit. We would like to thank other members of the PMIx community, and Ralph Castain in particular, for the excellent work that we build upon. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work is also supported by the ExaWorks project (part of the Exascale Computing Project (ECP)) under DOE Contract No. DE-SC0012704 and by the DOE HEP Center for Computational Excellence at Brookhaven National Laboratory under B&R KA2401045. We also acknowledge DOE INCITE awards for allocations on Summit.

Keywords

High performance computing
Middleware
Resource management
Runtime environment
Runtime system

Access to Document

10.1007/978-3-031-22698-4_5

Cite this

Titov, M., Turilli, M., Merzky, A., Naughton, T., Elwasif, W., & Jha, S. (2023). RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources. In D. Klusácek, C. Julita, & G. P. Rodrigo (Eds.), Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers (pp. 88-107). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13592 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-22698-4_5

Titov, Mikhail ; Turilli, Matteo ; Merzky, Andre et al. / RADICAL-Pilot and PMIx/PRRTE : Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources. Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers. editor / Dalibor Klusácek ; Corbalán Julita ; Gonzalo P. Rodrigo. Springer Science and Business Media Deutschland GmbH, 2023. pp. 88-107 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{a2eb3dff625345b49cf1706b7333b2c2,

title = "RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources",

abstract = "Execution of heterogeneous workflows on high-performance computing (HPC) platforms present unprecedented resource management and execution coordination challenges for runtime systems. Task heterogeneity increases the complexity of resource and execution management, limiting the scalability and efficiency of workflow execution. Resource partitioning and distribution of tasks execution over portioned resources promises to address those problems but we lack an experimental evaluation of its performance at scale. This paper provides a performance evaluation of the Process Management Interface for Exascale (PMIx) and its reference implementation PRRTE on the leadership-class HPC platform Summit, when integrated into a pilot-based runtime system called RADICAL-Pilot. We partition resources across multiple PRRTE Distributed Virtual Machine (DVM) environments, responsible for launching tasks via the PMIx interface. We experimentally measure the workload execution performance in terms of task scheduling/launching rate and distribution of DVM task placement times, DVM startup and termination overheads on the Summit leadership-class HPC platform. Integrated solution with PMIx/PRRTE enables using an abstracted, standardized set of interfaces for orchestrating the launch process, dynamic process management and monitoring capabilities. It extends scaling capabilities allowing to overcome a limitation of other launching mechanisms (e.g., JSM/LSF). Explored different DVM setup configurations provide insights on DVM performance and a layout to leverage it. Our experimental results show that heterogeneous workload of 65,500 tasks on 2048 nodes, and partitioned across 32 DVMs, runs steady with resource utilization not lower than 52 %. While having less concurrently executed tasks resource utilization is able to reach up to 85 %, based on results of heterogeneous workload of 8200 tasks on 256 nodes and 2 DVMs.",

keywords = "High performance computing, Middleware, Resource management, Runtime environment, Runtime system",

author = "Mikhail Titov and Matteo Turilli and Andre Merzky and Thomas Naughton and Wael Elwasif and Shantenu Jha",

note = "Publisher Copyright: {\textcopyright} 2023, The Author(s), under exclusive license to Springer Nature Switzerland AG.; 25th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2022, held in conjunction with the 36th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2022 ; Conference date: 03-06-2022 Through 03-06-2022",

year = "2023",

doi = "10.1007/978-3-031-22698-4_5",

language = "English",

isbn = "9783031226977",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "88--107",

editor = "Dalibor Klus{\'a}cek and Corbal{\'a}n Julita and Rodrigo, {Gonzalo P.}",

booktitle = "Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers",

}

Titov, M, Turilli, M, Merzky, A, Naughton, T , Elwasif, W & Jha, S 2023, RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources. in D Klusácek, C Julita & GP Rodrigo (eds), Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 13592 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 88-107, 25th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2022, held in conjunction with the 36th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2022, Virtual, Online, 06/3/22. https://doi.org/10.1007/978-3-031-22698-4_5

RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources. / Titov, Mikhail; Turilli, Matteo; Merzky, Andre et al.
Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers. ed. / Dalibor Klusácek; Corbalán Julita; Gonzalo P. Rodrigo. Springer Science and Business Media Deutschland GmbH, 2023. p. 88-107 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 13592 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - RADICAL-Pilot and PMIx/PRRTE

T2 - 25th International Workshop on Job Scheduling Strategies for Parallel Processing, JSSPP 2022, held in conjunction with the 36th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2022

AU - Titov, Mikhail

AU - Turilli, Matteo

AU - Merzky, Andre

AU - Naughton, Thomas

AU - Elwasif, Wael

AU - Jha, Shantenu

PY - 2023

Y1 - 2023

N2 - Execution of heterogeneous workflows on high-performance computing (HPC) platforms present unprecedented resource management and execution coordination challenges for runtime systems. Task heterogeneity increases the complexity of resource and execution management, limiting the scalability and efficiency of workflow execution. Resource partitioning and distribution of tasks execution over portioned resources promises to address those problems but we lack an experimental evaluation of its performance at scale. This paper provides a performance evaluation of the Process Management Interface for Exascale (PMIx) and its reference implementation PRRTE on the leadership-class HPC platform Summit, when integrated into a pilot-based runtime system called RADICAL-Pilot. We partition resources across multiple PRRTE Distributed Virtual Machine (DVM) environments, responsible for launching tasks via the PMIx interface. We experimentally measure the workload execution performance in terms of task scheduling/launching rate and distribution of DVM task placement times, DVM startup and termination overheads on the Summit leadership-class HPC platform. Integrated solution with PMIx/PRRTE enables using an abstracted, standardized set of interfaces for orchestrating the launch process, dynamic process management and monitoring capabilities. It extends scaling capabilities allowing to overcome a limitation of other launching mechanisms (e.g., JSM/LSF). Explored different DVM setup configurations provide insights on DVM performance and a layout to leverage it. Our experimental results show that heterogeneous workload of 65,500 tasks on 2048 nodes, and partitioned across 32 DVMs, runs steady with resource utilization not lower than 52 %. While having less concurrently executed tasks resource utilization is able to reach up to 85 %, based on results of heterogeneous workload of 8200 tasks on 256 nodes and 2 DVMs.

AB - Execution of heterogeneous workflows on high-performance computing (HPC) platforms present unprecedented resource management and execution coordination challenges for runtime systems. Task heterogeneity increases the complexity of resource and execution management, limiting the scalability and efficiency of workflow execution. Resource partitioning and distribution of tasks execution over portioned resources promises to address those problems but we lack an experimental evaluation of its performance at scale. This paper provides a performance evaluation of the Process Management Interface for Exascale (PMIx) and its reference implementation PRRTE on the leadership-class HPC platform Summit, when integrated into a pilot-based runtime system called RADICAL-Pilot. We partition resources across multiple PRRTE Distributed Virtual Machine (DVM) environments, responsible for launching tasks via the PMIx interface. We experimentally measure the workload execution performance in terms of task scheduling/launching rate and distribution of DVM task placement times, DVM startup and termination overheads on the Summit leadership-class HPC platform. Integrated solution with PMIx/PRRTE enables using an abstracted, standardized set of interfaces for orchestrating the launch process, dynamic process management and monitoring capabilities. It extends scaling capabilities allowing to overcome a limitation of other launching mechanisms (e.g., JSM/LSF). Explored different DVM setup configurations provide insights on DVM performance and a layout to leverage it. Our experimental results show that heterogeneous workload of 65,500 tasks on 2048 nodes, and partitioned across 32 DVMs, runs steady with resource utilization not lower than 52 %. While having less concurrently executed tasks resource utilization is able to reach up to 85 %, based on results of heterogeneous workload of 8200 tasks on 256 nodes and 2 DVMs.

KW - High performance computing

KW - Middleware

KW - Resource management

KW - Runtime environment

KW - Runtime system

UR - http://www.scopus.com/inward/record.url?scp=85148692928&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-22698-4_5

DO - 10.1007/978-3-031-22698-4_5

M3 - Conference contribution

AN - SCOPUS:85148692928

SN - 9783031226977

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 88

EP - 107

BT - Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers

A2 - Klusácek, Dalibor

A2 - Julita, Corbalán

A2 - Rodrigo, Gonzalo P.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 3 June 2022 through 3 June 2022

ER -

Titov M, Turilli M, Merzky A, Naughton T , Elwasif W, Jha S. RADICAL-Pilot and PMIx/PRRTE: Executing Heterogeneous Workloads at Large Scale on Partitioned HPC Resources. In Klusácek D, Julita C, Rodrigo GP, editors, Job Scheduling Strategies for Parallel Processing - 25th International Workshop, JSSPP 2022, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2023. p. 88-107. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-22698-4_5