MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

Justin M. Wozniak; Matthieu Dorier; Robert Ross; Tong Shu; Tahsin Kurc; Li Tang; Norbert Podhorszki; Matthew Wolf

doi:10.1016/j.future.2019.05.020

MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

Justin M. Wozniak, Matthieu Dorier, Robert Ross, Tong Shu, Tahsin Kurc, Li Tang, Norbert Podhorszki, Matthew Wolf

Workflow Systems

Research output: Contribution to journal › Article › peer-review

10 Scopus citations

Abstract

While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent's CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. We report results showing 2× throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

Original language	English
Pages (from-to)	576-589
Number of pages	14
Journal	Future Generation Computer Systems
Volume	101
DOIs	https://doi.org/10.1016/j.future.2019.05.020
State	Published - Dec 2019

Funding

We thank Pavan Balaji and Rajeev Thakur for their insights about the MPI standard and its implementations, MisbahMubarak for her help setting up CODES simulations, and the ROSS developers for quickly reacting to our feature requests. This research was supported by the Exascale Computing Project (17-SC-20-SC), a joint project of the U.S. Department of Energy’s Office of Science and National Nuclear Security Administration, responsible for delivering a capable exascale ecosystem, including software, applications, and hardware technology, to support the nation’s exascale computing imperative. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research , under contract number DE-AC02-06CH11357 We gratefully acknowledge the computing resources provided on Blues, a high-performance computing cluster operated by the Laboratory Computing Resource Center at Argonne National Laboratory. Experiments presented in this paper were carried out using the Grid’5000 testbed, supported by a scientific interest group hosted by Inria and including CNRS, RENATER and several universities as well as other organizations (see https://www.grid5000.fr ).

Funders	Funder number
Laboratory Computing Resource Center
U.S. Department of Energy
Office of Science
National Nuclear Security Administration
Advanced Scientific Computing Research	DE-AC02-06CH11357
Argonne National Laboratory

Keywords

Cram
Ensemble simulations
MPI
MPI_Comm_launch
MPMD
Swift/T
Workflows

Access to Document

10.1016/j.future.2019.05.020

Cite this

@article{2b8c06cf397f4da0af9d4e000d48c929,

title = "MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows",

abstract = "While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent's CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. We report results showing 2× throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.",

keywords = "Cram, Ensemble simulations, MPI, MPI_Comm_launch, MPMD, Swift/T, Workflows",

author = "Wozniak, {Justin M.} and Matthieu Dorier and Robert Ross and Tong Shu and Tahsin Kurc and Li Tang and Norbert Podhorszki and Matthew Wolf",

note = "Publisher Copyright: {\textcopyright} 2019",

year = "2019",

month = dec,

doi = "10.1016/j.future.2019.05.020",

language = "English",

volume = "101",

pages = "576--589",

journal = "Future Generation Computer Systems",

issn = "0167-739X",

publisher = "Elsevier",

}

TY - JOUR

T1 - MPI jobs within MPI jobs

T2 - A practical way of enabling task-level fault-tolerance in HPC workflows

AU - Wozniak, Justin M.

AU - Dorier, Matthieu

AU - Ross, Robert

AU - Shu, Tong

AU - Kurc, Tahsin

AU - Tang, Li

AU - Podhorszki, Norbert

AU - Wolf, Matthew

PY - 2019/12

Y1 - 2019/12

N2 - While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent's CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. We report results showing 2× throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

AB - While the use of workflows for HPC is growing, MPI interoperability remains a challenge for workflow management systems. The MPI standard and/or its implementations provide a number of ways to build multiple-programs-multiple-data (MPMD) applications. These methods present limitations related to fault tolerance, and are not easy to use. In this paper, we advocate for a novel MPI_Comm_launch function acting as the parallel counterpart of a system(3) call. MPI_Comm_launch allows a child MPI application to be launched inside the resources originally held by processes of a parent MPI application. Two important aspects of MPI_Comm_launch is that it pauses the calling process, and runs the child processes on the parent's CPU cores, but in an isolated manner with respect to memory. This function makes it easier to build MPMD applications with well-decoupled subtasks. We show how this feature can provide better flexibility and better fault tolerance in ensemble simulations and HPC workflows. We report results showing 2× throughput improvement for application workflows with faults, and scaling results for challenging workloads up to 256 nodes.

KW - Cram

KW - Ensemble simulations

KW - MPI

KW - MPI_Comm_launch

KW - MPMD

KW - Swift/T

KW - Workflows

UR - http://www.scopus.com/inward/record.url?scp=85068512474&partnerID=8YFLogxK

U2 - 10.1016/j.future.2019.05.020

DO - 10.1016/j.future.2019.05.020

M3 - Article

AN - SCOPUS:85068512474

SN - 0167-739X

VL - 101

SP - 576

EP - 589

JO - Future Generation Computer Systems

JF - Future Generation Computer Systems

ER -

MPI jobs within MPI jobs: A practical way of enabling task-level fault-tolerance in HPC workflows

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this