Gpu lifetimes on titan supercomputer: Survival analysis and reliability

George Ostrouchov; Don Maxwell; Rizwan A. Ashraf; Christian Engelmann; Mallikarjun Shankar; James H. Rogers

doi:10.1109/SC41405.2020.00045

Gpu lifetimes on titan supercomputer: Survival analysis and reliability

George Ostrouchov, Don Maxwell, Rizwan A. Ashraf, Christian Engelmann, Mallikarjun Shankar, James H. Rogers

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

20 Scopus citations

Abstract

The Cray XK7 Titan was the top supercomputer system in the world for a long time and remained critically important throughout its nearly seven year life. It was an interesting machine from a reliability viewpoint as most of its power came from 18,688 GPUs whose operation was forced to execute three rework cycles, two on the GPU mechanical assembly and one on the GPU circuitboards. We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan's 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling. We describe the history, data collection, cleaning, and analysis and give recommendations for future supercomputing systems. We make the data and our analysis codes publicly available.

Original language	English
Title of host publication	Proceedings of SC 2020
Subtitle of host publication	International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	IEEE Computer Society
ISBN (Electronic)	9781728199986
DOIs	https://doi.org/10.1109/SC41405.2020.00045
State	Published - Nov 2020
Event	2020 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020 - Virtual, Atlanta, United States Duration: Nov 9 2020 → Nov 19 2020

Publication series

Name	International Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume	2020-November
ISSN (Print)	2167-4329
ISSN (Electronic)	2167-4337

Conference

Conference	2020 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020
Country/Territory	United States
City	Virtual, Atlanta
Period	11/9/20 → 11/19/20

Funding

This work was sponsored by the U.S. Department of Energy’s Office of Advanced Scientific Computing Research. This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work was supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Resilience for Extreme Scale Supercomputing Systems Program, with program managers Robinson Pino and Lucy Nowell.

Funders	Funder number
Advanced Scientific Computing Research	DE-AC05-00OR22725
U.S. Department of Energy
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science
Advanced Scientific Computing Research
U.S. Department of Energy
Office of Science

Keywords

Cox regression
Cray
GPU
GPU failure data set
Kaplan-Meier survival
MTBF
NVIDIA
large-scale systems
log analysis
reliability
supercomputer

Access to Document

10.1109/SC41405.2020.00045

Cite this

Ostrouchov, G., Maxwell, D., Ashraf, R. A., Engelmann, C., Shankar, M., & Rogers, J. H. (2020). Gpu lifetimes on titan supercomputer: Survival analysis and reliability. In Proceedings of SC 2020: International Conference for High Performance Computing, Networking, Storage and Analysis Article 9355319 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 2020-November). IEEE Computer Society. https://doi.org/10.1109/SC41405.2020.00045

Ostrouchov, George ; Maxwell, Don ; Ashraf, Rizwan A. et al. / Gpu lifetimes on titan supercomputer : Survival analysis and reliability. Proceedings of SC 2020: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2020. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

@inproceedings{20eb37acaa294dabbcced656ccae1f9e,

title = "Gpu lifetimes on titan supercomputer: Survival analysis and reliability",

abstract = "The Cray XK7 Titan was the top supercomputer system in the world for a long time and remained critically important throughout its nearly seven year life. It was an interesting machine from a reliability viewpoint as most of its power came from 18,688 GPUs whose operation was forced to execute three rework cycles, two on the GPU mechanical assembly and one on the GPU circuitboards. We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan's 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling. We describe the history, data collection, cleaning, and analysis and give recommendations for future supercomputing systems. We make the data and our analysis codes publicly available.",

keywords = "Cox regression, Cray, GPU, GPU failure data set, Kaplan-Meier survival, MTBF, NVIDIA, large-scale systems, log analysis, reliability, supercomputer",

author = "George Ostrouchov and Don Maxwell and Ashraf, {Rizwan A.} and Christian Engelmann and Mallikarjun Shankar and Rogers, {James H.}",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020 ; Conference date: 09-11-2020 Through 19-11-2020",

year = "2020",

month = nov,

doi = "10.1109/SC41405.2020.00045",

language = "English",

series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",

publisher = "IEEE Computer Society",

booktitle = "Proceedings of SC 2020",

}

Ostrouchov, G, Maxwell, D, Ashraf, RA, Engelmann, C , Shankar, M & Rogers, JH 2020, Gpu lifetimes on titan supercomputer: Survival analysis and reliability. in Proceedings of SC 2020: International Conference for High Performance Computing, Networking, Storage and Analysis., 9355319, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, vol. 2020-November, IEEE Computer Society, 2020 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020, Virtual, Atlanta, United States, 11/9/20. https://doi.org/10.1109/SC41405.2020.00045

Gpu lifetimes on titan supercomputer: Survival analysis and reliability. / Ostrouchov, George; Maxwell, Don; Ashraf, Rizwan A. et al.
Proceedings of SC 2020: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2020. 9355319 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 2020-November).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Gpu lifetimes on titan supercomputer

T2 - 2020 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2020

AU - Ostrouchov, George

AU - Maxwell, Don

AU - Ashraf, Rizwan A.

AU - Engelmann, Christian

AU - Shankar, Mallikarjun

AU - Rogers, James H.

PY - 2020/11

Y1 - 2020/11

N2 - The Cray XK7 Titan was the top supercomputer system in the world for a long time and remained critically important throughout its nearly seven year life. It was an interesting machine from a reliability viewpoint as most of its power came from 18,688 GPUs whose operation was forced to execute three rework cycles, two on the GPU mechanical assembly and one on the GPU circuitboards. We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan's 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling. We describe the history, data collection, cleaning, and analysis and give recommendations for future supercomputing systems. We make the data and our analysis codes publicly available.

AB - The Cray XK7 Titan was the top supercomputer system in the world for a long time and remained critically important throughout its nearly seven year life. It was an interesting machine from a reliability viewpoint as most of its power came from 18,688 GPUs whose operation was forced to execute three rework cycles, two on the GPU mechanical assembly and one on the GPU circuitboards. We write about the last rework cycle and a reliability analysis of over 100,000 years of GPU lifetimes during Titan's 6-year-long productive period. Using time between failures analysis and statistical survival analysis techniques, we find that GPU reliability is dependent on heat dissipation to an extent that strongly correlates with detailed nuances of the cooling architecture and job scheduling. We describe the history, data collection, cleaning, and analysis and give recommendations for future supercomputing systems. We make the data and our analysis codes publicly available.

KW - Cox regression

KW - Cray

KW - GPU

KW - GPU failure data set

KW - Kaplan-Meier survival

KW - MTBF

KW - NVIDIA

KW - large-scale systems

KW - log analysis

KW - reliability

KW - supercomputer

UR - http://www.scopus.com/inward/record.url?scp=85099562744&partnerID=8YFLogxK

U2 - 10.1109/SC41405.2020.00045

DO - 10.1109/SC41405.2020.00045

M3 - Conference contribution

AN - SCOPUS:85099562744

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2020

PB - IEEE Computer Society

Y2 - 9 November 2020 through 19 November 2020

ER -

Ostrouchov G, Maxwell D, Ashraf RA, Engelmann C , Shankar M , Rogers JH. Gpu lifetimes on titan supercomputer: Survival analysis and reliability. In Proceedings of SC 2020: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2020. 9355319. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). doi: 10.1109/SC41405.2020.00045

Gpu lifetimes on titan supercomputer: Survival analysis and reliability

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint