Orchestrating Fault Prediction with Live Migration and Checkpointing

Subhendu Behera; Lipeng Wan; Frank Mueller; Matthew Wolf; Scott Klasky

doi:10.1145/3369583.3392672

Orchestrating Fault Prediction with Live Migration and Checkpointing

Subhendu Behera, Lipeng Wan, Frank Mueller, Matthew Wolf, Scott Klasky

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

8 Scopus citations

Abstract

Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ∼20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ∼29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

Original language	English
Title of host publication	HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing
Publisher	Association for Computing Machinery, Inc
Pages	167-171
Number of pages	5
ISBN (Electronic)	9781450370523
DOIs	https://doi.org/10.1145/3369583.3392672
State	Published - Jun 23 2020
Event	29th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2020 - Stockholm, Sweden Duration: Jun 23 2020 → Jun 26 2020

Publication series

Name	HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing

Conference

Conference	29th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2020
Country/Territory	Sweden
City	Stockholm
Period	06/23/20 → 06/26/20

Funding

This research was supported in part by NSF grants 1525609, 1813004, and an appointment to the Oak Ridge National Laboratory ASTRO Program, sponsored by the U.S. Department of Energy and administered by the Oak Ridge Institute for Science and Education. This research was also supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Funders	Funder number
U.S. Department of Energy Office of Science
National Science Foundation	1813004, 1525609
National Science Foundation
U.S. Department of Energy	DE-AC05-00OR22725
U.S. Department of Energy
Office of Science
National Nuclear Security Administration
Oak Ridge Institute for Science and Education	17-SC-20-SC
Oak Ridge Institute for Science and Education

Keywords

I/O subsystem
burst buffers
checkpoint/restart
failure prediction
high performance computing
proactive live migration

Access to Document

10.1145/3369583.3392672

Cite this

Behera, S., Wan, L., Mueller, F., Wolf, M., & Klasky, S. (2020). Orchestrating Fault Prediction with Live Migration and Checkpointing. In HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing (pp. 167-171). (HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/3369583.3392672

Behera, Subhendu ; Wan, Lipeng ; Mueller, Frank et al. / Orchestrating Fault Prediction with Live Migration and Checkpointing. HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc, 2020. pp. 167-171 (HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing).

@inproceedings{0b1469185995435ea0741b7cf54d5286,

title = "Orchestrating Fault Prediction with Live Migration and Checkpointing",

abstract = "Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ∼20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ∼29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.",

keywords = "I/O subsystem, burst buffers, checkpoint/restart, failure prediction, high performance computing, proactive live migration",

author = "Subhendu Behera and Lipeng Wan and Frank Mueller and Matthew Wolf and Scott Klasky",

note = "Publisher Copyright: {\textcopyright} 2020 ACM.; 29th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2020 ; Conference date: 23-06-2020 Through 26-06-2020",

year = "2020",

month = jun,

day = "23",

doi = "10.1145/3369583.3392672",

language = "English",

series = "HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing",

publisher = "Association for Computing Machinery, Inc",

pages = "167--171",

booktitle = "HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing",

}

Behera, S, Wan, L, Mueller, F, Wolf, M & Klasky, S 2020, Orchestrating Fault Prediction with Live Migration and Checkpointing. in HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing, Association for Computing Machinery, Inc, pp. 167-171, 29th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2020, Stockholm, Sweden, 06/23/20. https://doi.org/10.1145/3369583.3392672

Orchestrating Fault Prediction with Live Migration and Checkpointing. / Behera, Subhendu; Wan, Lipeng; Mueller, Frank et al.
HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc, 2020. p. 167-171 (HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Orchestrating Fault Prediction with Live Migration and Checkpointing

AU - Behera, Subhendu

AU - Wan, Lipeng

AU - Mueller, Frank

AU - Wolf, Matthew

AU - Klasky, Scott

PY - 2020/6/23

Y1 - 2020/6/23

N2 - Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ∼20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ∼29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

AB - Checkpoint/Restart (C/R) is widely used to provide fault tolerance on High-Performance Computing (HPC) systems. However, Parallel File System (PFS) overhead and failure uncertainty cause significant application overhead. This paper develops an adaptive multi-level C/R model that incorporates a failure prediction and analysis model, which orchestrates failure prediction, checkpointing, checkpoint frequency, and proactive live migration along with the additional benefit of Burst Buffers (BB). It effectively reduces the overheads due to failures, checkpointing, and recovery. Simulation results for the Summit supercomputer yield a reduction of ∼20%-86% in application overhead due to BBs, orchestrated failure prediction, and migration. We also observe a ∼29% decrease in checkpoint writes to BBs, which can increase the longevity of the BB storage devices.

KW - I/O subsystem

KW - burst buffers

KW - checkpoint/restart

KW - failure prediction

KW - high performance computing

KW - proactive live migration

UR - http://www.scopus.com/inward/record.url?scp=85088359263&partnerID=8YFLogxK

U2 - 10.1145/3369583.3392672

DO - 10.1145/3369583.3392672

M3 - Conference contribution

AN - SCOPUS:85088359263

T3 - HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing

SP - 167

EP - 171

BT - HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing

PB - Association for Computing Machinery, Inc

T2 - 29th International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2020

Y2 - 23 June 2020 through 26 June 2020

ER -

Behera S, Wan L, Mueller F, Wolf M, Klasky S. Orchestrating Fault Prediction with Live Migration and Checkpointing. In HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc. 2020. p. 167-171. (HPDC 2020 - Proceedings of the 29th International Symposium on High-Performance Parallel and Distributed Computing). doi: 10.1145/3369583.3392672

Orchestrating Fault Prediction with Live Migration and Checkpointing

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this