Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility

Devesh Tiwari; Saurabh Gupta; George Gallarno; Jim Rogers; Don Maxwell

doi:10.1145/2807591.2807666

Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility

Devesh Tiwari, Saurabh Gupta, George Gallarno, Jim Rogers, Don Maxwell

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

67 Scopus citations

Abstract

The high computational capability of graphics processing units (GPUs) is enabling and driving the scientific discovery process at large-scale. The world's second fastest supercomputer for open science, Titan, has more than 18,000 GPUs that computational scientists use to perform scientific simulations and data analysis. Understanding of GPU reliability characteristics, however, is still in its nascent stage since GPUs have only recently been deployed at large-scale. This paper presents a detailed study of GPU errors and their impact on system operations and applications, describing experiences with the 18,688 GPUs on the Titan supercomputer as well as lessons learned in the process of efficient operation of GPUs at scale. These experiences are helpful to HPC sites which already have large-scale GPU clusters or plan to deploy GPUs in the future.

Original language	English
Title of host publication	Proceedings of SC 2015
Subtitle of host publication	The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	IEEE Computer Society
ISBN (Electronic)	9781450337236
DOIs	https://doi.org/10.1145/2807591.2807666
State	Published - Nov 15 2015
Event	International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015 - Austin, United States Duration: Nov 15 2015 → Nov 20 2015

Publication series

Name	International Conference for High Performance Computing, Networking, Storage and Analysis, SC
Volume	15-20-November-2015
ISSN (Print)	2167-4329
ISSN (Electronic)	2167-4337

Conference

Conference	International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015
Country/Territory	United States
City	Austin
Period	11/15/15 → 11/20/15

Funding

We thank the reviewers for their feedback that has significantly improved the paper. George Gallarno was supported by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the SULI Program. This work was supported by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is managed by UT Battelle, LLC for the U.S. DOE (under the contract No. DE-AC05-00OR22725).

Access to Document

10.1145/2807591.2807666

Cite this

Tiwari, D., Gupta, S., Gallarno, G., Rogers, J., & Maxwell, D. (2015). Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility. In Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis Article a38 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 15-20-November-2015). IEEE Computer Society. https://doi.org/10.1145/2807591.2807666

Tiwari, Devesh ; Gupta, Saurabh ; Gallarno, George et al. / Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility. Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2015. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

@inproceedings{a53e33c6d7714320b10c3b5cd4bda474,

title = "Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility",

abstract = "The high computational capability of graphics processing units (GPUs) is enabling and driving the scientific discovery process at large-scale. The world's second fastest supercomputer for open science, Titan, has more than 18,000 GPUs that computational scientists use to perform scientific simulations and data analysis. Understanding of GPU reliability characteristics, however, is still in its nascent stage since GPUs have only recently been deployed at large-scale. This paper presents a detailed study of GPU errors and their impact on system operations and applications, describing experiences with the 18,688 GPUs on the Titan supercomputer as well as lessons learned in the process of efficient operation of GPUs at scale. These experiences are helpful to HPC sites which already have large-scale GPU clusters or plan to deploy GPUs in the future.",

author = "Devesh Tiwari and Saurabh Gupta and George Gallarno and Jim Rogers and Don Maxwell",

note = "Publisher Copyright: {\textcopyright} 2015 ACM.; International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015 ; Conference date: 15-11-2015 Through 20-11-2015",

year = "2015",

month = nov,

day = "15",

doi = "10.1145/2807591.2807666",

language = "English",

series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",

publisher = "IEEE Computer Society",

booktitle = "Proceedings of SC 2015",

}

Tiwari, D, Gupta, S, Gallarno, G, Rogers, J & Maxwell, D 2015, Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility. in Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis., a38, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, vol. 15-20-November-2015, IEEE Computer Society, International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015, Austin, United States, 11/15/15. https://doi.org/10.1145/2807591.2807666

Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility. / Tiwari, Devesh; Gupta, Saurabh; Gallarno, George et al.
Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2015. a38 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC; Vol. 15-20-November-2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility

AU - Tiwari, Devesh

AU - Gupta, Saurabh

AU - Gallarno, George

AU - Rogers, Jim

AU - Maxwell, Don

PY - 2015/11/15

Y1 - 2015/11/15

N2 - The high computational capability of graphics processing units (GPUs) is enabling and driving the scientific discovery process at large-scale. The world's second fastest supercomputer for open science, Titan, has more than 18,000 GPUs that computational scientists use to perform scientific simulations and data analysis. Understanding of GPU reliability characteristics, however, is still in its nascent stage since GPUs have only recently been deployed at large-scale. This paper presents a detailed study of GPU errors and their impact on system operations and applications, describing experiences with the 18,688 GPUs on the Titan supercomputer as well as lessons learned in the process of efficient operation of GPUs at scale. These experiences are helpful to HPC sites which already have large-scale GPU clusters or plan to deploy GPUs in the future.

AB - The high computational capability of graphics processing units (GPUs) is enabling and driving the scientific discovery process at large-scale. The world's second fastest supercomputer for open science, Titan, has more than 18,000 GPUs that computational scientists use to perform scientific simulations and data analysis. Understanding of GPU reliability characteristics, however, is still in its nascent stage since GPUs have only recently been deployed at large-scale. This paper presents a detailed study of GPU errors and their impact on system operations and applications, describing experiences with the 18,688 GPUs on the Titan supercomputer as well as lessons learned in the process of efficient operation of GPUs at scale. These experiences are helpful to HPC sites which already have large-scale GPU clusters or plan to deploy GPUs in the future.

UR - http://www.scopus.com/inward/record.url?scp=84966670278&partnerID=8YFLogxK

U2 - 10.1145/2807591.2807666

DO - 10.1145/2807591.2807666

M3 - Conference contribution

AN - SCOPUS:84966670278

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2015

PB - IEEE Computer Society

T2 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2015

Y2 - 15 November 2015 through 20 November 2015

ER -

Tiwari D, Gupta S, Gallarno G, Rogers J , Maxwell D. Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility. In Proceedings of SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2015. a38. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). doi: 10.1145/2807591.2807666

Reliability lessons learned from GPU experience with the Titan supercomputer at Oak Ridge leadership computing facility

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this