Machine learning models for GPU error prediction in a large scale HPC system

Bin Nie; Ji Xue; Saurabh Gupta; Tirthak Patel; Christian Engelmann; Evgenia Smirni; Devesh Tiwari

doi:10.1109/DSN.2018.00022

Machine learning models for GPU error prediction in a large scale HPC system

Bin Nie, Ji Xue, Saurabh Gupta, Tirthak Patel, Christian Engelmann, Evgenia Smirni, Devesh Tiwari

Intelligent Systems and Facilities

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

60 Scopus citations

Abstract

GPUs are widely deployed on large-scale HPC systems to provide powerful computational capability for scientific applications from various domains. As those applications are normally long-running, investigating the characteristics of GPU errors becomes imperative for reliability. In this paper, we first study the system conditions that trigger GPU errors using six-month trace data collected from a large-scale, operational HPC system. Then, we use machine learning to predict the occurrence of GPU errors, by taking advantage of temporal and spatial dependencies of the trace data. The resulting machine learning prediction framework is robust and accurate under different workloads.

Original language	English
Title of host publication	Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	95-106
Number of pages	12
ISBN (Electronic)	9781538655955
DOIs	https://doi.org/10.1109/DSN.2018.00022
State	Published - Jul 19 2018
Event	48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 - Luxembourg City, Luxembourg Duration: Jun 25 2018 → Jun 28 2018

Publication series

Name	Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

Conference

Conference	48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018
Country/Territory	Luxembourg
City	Luxembourg City
Period	06/25/18 → 06/28/18

Funding

Acknowledgment We thank reviewers for their constructive feedback. The work was supported by in part through NSF grants CCF-1649087, CCF-1717532, Northeastern University, and by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, program manager Lucy Nowell. This work also used in part the resources of, the Oak Ridge Leadership Computing Facility, located in the National Center for Computational Sciences at ORNL, which is managed by UT Battelle, LLC for the U.S. DOE under contract number DE-AC05-00OR22725. This manuscript has been authored by UT-Battelle,LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Funders	Funder number
National Science Foundation	1717532, CCF-1649087, CCF-1717532, 1649087
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science
Advanced Scientific Computing Research
Northeastern University

Keywords

Error Prediction
GPU Reliability
HPC
Machine Learning
System Reliability

Access to Document

10.1109/DSN.2018.00022

Cite this

Nie, B., Xue, J., Gupta, S., Patel, T., Engelmann, C., Smirni, E., & Tiwari, D. (2018). Machine learning models for GPU error prediction in a large scale HPC system. In Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 (pp. 95-106). Article 8416474 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DSN.2018.00022

Nie, Bin ; Xue, Ji ; Gupta, Saurabh et al. / Machine learning models for GPU error prediction in a large scale HPC system. Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 95-106 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018).

@inproceedings{765edb5722d449168e3c089933f173a5,

title = "Machine learning models for GPU error prediction in a large scale HPC system",

abstract = "GPUs are widely deployed on large-scale HPC systems to provide powerful computational capability for scientific applications from various domains. As those applications are normally long-running, investigating the characteristics of GPU errors becomes imperative for reliability. In this paper, we first study the system conditions that trigger GPU errors using six-month trace data collected from a large-scale, operational HPC system. Then, we use machine learning to predict the occurrence of GPU errors, by taking advantage of temporal and spatial dependencies of the trace data. The resulting machine learning prediction framework is robust and accurate under different workloads.",

keywords = "Error Prediction, GPU Reliability, HPC, Machine Learning, System Reliability",

author = "Bin Nie and Ji Xue and Saurabh Gupta and Tirthak Patel and Christian Engelmann and Evgenia Smirni and Devesh Tiwari",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 ; Conference date: 25-06-2018 Through 28-06-2018",

year = "2018",

month = jul,

day = "19",

doi = "10.1109/DSN.2018.00022",

language = "English",

series = "Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "95--106",

booktitle = "Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018",

}

Nie, B, Xue, J, Gupta, S, Patel, T, Engelmann, C, Smirni, E & Tiwari, D 2018, Machine learning models for GPU error prediction in a large scale HPC system. in Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018., 8416474, Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Institute of Electrical and Electronics Engineers Inc., pp. 95-106, 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Luxembourg City, Luxembourg, 06/25/18. https://doi.org/10.1109/DSN.2018.00022

Machine learning models for GPU error prediction in a large scale HPC system. / Nie, Bin; Xue, Ji; Gupta, Saurabh et al.
Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 95-106 8416474 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Machine learning models for GPU error prediction in a large scale HPC system

AU - Nie, Bin

AU - Xue, Ji

AU - Gupta, Saurabh

AU - Patel, Tirthak

AU - Engelmann, Christian

AU - Smirni, Evgenia

AU - Tiwari, Devesh

PY - 2018/7/19

Y1 - 2018/7/19

N2 - GPUs are widely deployed on large-scale HPC systems to provide powerful computational capability for scientific applications from various domains. As those applications are normally long-running, investigating the characteristics of GPU errors becomes imperative for reliability. In this paper, we first study the system conditions that trigger GPU errors using six-month trace data collected from a large-scale, operational HPC system. Then, we use machine learning to predict the occurrence of GPU errors, by taking advantage of temporal and spatial dependencies of the trace data. The resulting machine learning prediction framework is robust and accurate under different workloads.

AB - GPUs are widely deployed on large-scale HPC systems to provide powerful computational capability for scientific applications from various domains. As those applications are normally long-running, investigating the characteristics of GPU errors becomes imperative for reliability. In this paper, we first study the system conditions that trigger GPU errors using six-month trace data collected from a large-scale, operational HPC system. Then, we use machine learning to predict the occurrence of GPU errors, by taking advantage of temporal and spatial dependencies of the trace data. The resulting machine learning prediction framework is robust and accurate under different workloads.

KW - Error Prediction

KW - GPU Reliability

KW - HPC

KW - Machine Learning

KW - System Reliability

UR - http://www.scopus.com/inward/record.url?scp=85051090091&partnerID=8YFLogxK

U2 - 10.1109/DSN.2018.00022

DO - 10.1109/DSN.2018.00022

M3 - Conference contribution

AN - SCOPUS:85051090091

T3 - Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

SP - 95

EP - 106

BT - Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

Y2 - 25 June 2018 through 28 June 2018

ER -

Nie B, Xue J, Gupta S, Patel T, Engelmann C, Smirni E et al. Machine learning models for GPU error prediction in a large scale HPC system. In Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 95-106. 8416474. (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018). doi: 10.1109/DSN.2018.00022

Machine learning models for GPU error prediction in a large scale HPC system

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this