MCEM: Multi-level cooperative exception model for HPC workflows

Stephen Herbein; David Domyancic; Paul Minner; Ignacio Laguna; Rafael Ferreira Da Silva; Dong H. Ahn

doi:10.1145/3322789.3328745

MCEM: Multi-level cooperative exception model for HPC workflows

Stephen Herbein, David Domyancic, Paul Minner, Ignacio Laguna, Rafael Ferreira Da Silva, Dong H. Ahn

Workflow and Ecosystem Services

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

As fault recovery mechanisms become increasingly important in HPC systems, the need for a new recovery model for workflows on these systems grows as well. While the traditional approach in which each system component attempts its own independent recovery after a fault works well at each individual application level, this model does not scale to the new level demanded by workflow-level exception handling. As today's workflows must often run many components simultaneously (e.g., workflow manager components, many simulation instances, data analytics etc), any uncoordinated model can quickly result in redundant or contradictory recovery actions. In this paper, we propose a multi-level cooperative exception model (MCEM), a novel exception handling approach that solves this coordination challenge for HPC workflows. We present our model, describe how it can be applied to common system faults and other workflow specific exceptions, and demonstrate how it reduces redundant I/O in the case of a file-system quota exception.

Original language	English
Title of host publication	ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019
Publisher	Association for Computing Machinery, Inc
Pages	27-32
Number of pages	6
ISBN (Electronic)	9781450367554
DOIs	https://doi.org/10.1145/3322789.3328745
State	Published - Jun 17 2019
Event	9th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS 2019, co-located with HPDC 2019 - Phoenix, United States Duration: Jun 25 2019 → …

Publication series

Name	ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019

Conference

Conference	9th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS 2019, co-located with HPDC 2019
Country/Territory	United States
City	Phoenix
Period	06/25/19 → …

Funding

This article has been authored by Lawrence Livermore National Security, LLC under Contract No. DE-AC52-07NA27344 with the U.S. Department of Energy. Accordingly, the United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this article or allow others to do so, for United States Government purposes (LLNL-CONF-771601).

Keywords

Coordinated recovery
Fault-tolerance
Workflow exception model

Access to Document

10.1145/3322789.3328745

Cite this

Herbein, S., Domyancic, D., Minner, P., Laguna, I., Da Silva, R. F., & Ahn, D. H. (2019). MCEM: Multi-level cooperative exception model for HPC workflows. In ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019 (pp. 27-32). (ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3322789.3328745

Herbein, Stephen ; Domyancic, David ; Minner, Paul et al. / MCEM : Multi-level cooperative exception model for HPC workflows. ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019. Association for Computing Machinery, Inc, 2019. pp. 27-32 (ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019).

@inproceedings{10b85eb6af7f46799fa28af687e5299d,

title = "MCEM: Multi-level cooperative exception model for HPC workflows",

abstract = "As fault recovery mechanisms become increasingly important in HPC systems, the need for a new recovery model for workflows on these systems grows as well. While the traditional approach in which each system component attempts its own independent recovery after a fault works well at each individual application level, this model does not scale to the new level demanded by workflow-level exception handling. As today's workflows must often run many components simultaneously (e.g., workflow manager components, many simulation instances, data analytics etc), any uncoordinated model can quickly result in redundant or contradictory recovery actions. In this paper, we propose a multi-level cooperative exception model (MCEM), a novel exception handling approach that solves this coordination challenge for HPC workflows. We present our model, describe how it can be applied to common system faults and other workflow specific exceptions, and demonstrate how it reduces redundant I/O in the case of a file-system quota exception.",

keywords = "Coordinated recovery, Fault-tolerance, Workflow exception model",

author = "Stephen Herbein and David Domyancic and Paul Minner and Ignacio Laguna and \{Da Silva\}, \{Rafael Ferreira\} and Ahn, \{Dong H.\}",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computing Machinery.; 9th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS 2019, co-located with HPDC 2019 ; Conference date: 25-06-2019",

year = "2019",

month = jun,

day = "17",

doi = "10.1145/3322789.3328745",

language = "English",

series = "ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019",

publisher = "Association for Computing Machinery, Inc",

pages = "27--32",

booktitle = "ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019",

}

Herbein, S, Domyancic, D, Minner, P, Laguna, I, Da Silva, RF & Ahn, DH 2019, MCEM: Multi-level cooperative exception model for HPC workflows. in ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019. ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019, Association for Computing Machinery, Inc, pp. 27-32, 9th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS 2019, co-located with HPDC 2019, Phoenix, United States, 06/25/19. https://doi.org/10.1145/3322789.3328745

MCEM: Multi-level cooperative exception model for HPC workflows. / Herbein, Stephen; Domyancic, David; Minner, Paul et al.
ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019. Association for Computing Machinery, Inc, 2019. p. 27-32 (ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - MCEM

T2 - 9th International Workshop on Runtime and Operating Systems for Supercomputers, ROSS 2019, co-located with HPDC 2019

AU - Herbein, Stephen

AU - Domyancic, David

AU - Minner, Paul

AU - Laguna, Ignacio

AU - Da Silva, Rafael Ferreira

AU - Ahn, Dong H.

PY - 2019/6/17

Y1 - 2019/6/17

N2 - As fault recovery mechanisms become increasingly important in HPC systems, the need for a new recovery model for workflows on these systems grows as well. While the traditional approach in which each system component attempts its own independent recovery after a fault works well at each individual application level, this model does not scale to the new level demanded by workflow-level exception handling. As today's workflows must often run many components simultaneously (e.g., workflow manager components, many simulation instances, data analytics etc), any uncoordinated model can quickly result in redundant or contradictory recovery actions. In this paper, we propose a multi-level cooperative exception model (MCEM), a novel exception handling approach that solves this coordination challenge for HPC workflows. We present our model, describe how it can be applied to common system faults and other workflow specific exceptions, and demonstrate how it reduces redundant I/O in the case of a file-system quota exception.

AB - As fault recovery mechanisms become increasingly important in HPC systems, the need for a new recovery model for workflows on these systems grows as well. While the traditional approach in which each system component attempts its own independent recovery after a fault works well at each individual application level, this model does not scale to the new level demanded by workflow-level exception handling. As today's workflows must often run many components simultaneously (e.g., workflow manager components, many simulation instances, data analytics etc), any uncoordinated model can quickly result in redundant or contradictory recovery actions. In this paper, we propose a multi-level cooperative exception model (MCEM), a novel exception handling approach that solves this coordination challenge for HPC workflows. We present our model, describe how it can be applied to common system faults and other workflow specific exceptions, and demonstrate how it reduces redundant I/O in the case of a file-system quota exception.

KW - Coordinated recovery

KW - Fault-tolerance

KW - Workflow exception model

UR - https://www.scopus.com/pages/publications/85069219572

U2 - 10.1145/3322789.3328745

DO - 10.1145/3322789.3328745

M3 - Conference contribution

AN - SCOPUS:85069219572

T3 - ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019

SP - 27

EP - 32

BT - ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019

PB - Association for Computing Machinery, Inc

Y2 - 25 June 2019

ER -

Herbein S, Domyancic D, Minner P, Laguna I, Da Silva RF, Ahn DH. MCEM: Multi-level cooperative exception model for HPC workflows. In ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019. Association for Computing Machinery, Inc. 2019. p. 27-32. (ROSS 2019 - Proceedings of the 9th International Workshop on Runtime and Operating Systems for Supercomputers, co-located with HPDC 2019). doi: 10.1145/3322789.3328745

MCEM: Multi-level cooperative exception model for HPC workflows

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this