GPU age-aware scheduling to improve the reliability of leadership jobs on titan

Christopher Zimmer; Don Maxwell; Stephen McNally; Scott Atchley; Sudharshan S. Vazhkudai

doi:10.1109/SC.2018.00010

GPU age-aware scheduling to improve the reliability of leadership jobs on titan

Christopher Zimmer, Don Maxwell, Stephen McNally, Scott Atchley, Sudharshan S. Vazhkudai

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ∼50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.

Original language	English
Title of host publication	Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	83-93
Number of pages	11
ISBN (Electronic)	9781538683842
DOIs	https://doi.org/10.1109/SC.2018.00010
State	Published - Jul 2 2018
Event	2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 - Dallas, United States Duration: Nov 11 2018 → Nov 16 2018

Publication series

Name	Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

Conference

Conference	2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018
Country/Territory	United States
City	Dallas
Period	11/11/18 → 11/16/18

Funding

This work was supported by the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is managed by UT Battelle, LLC for the U.S. DOE (under the contract No. DE-AC05-00OR22725).

Access to Document

10.1109/SC.2018.00010

Cite this

Zimmer, C., Maxwell, D., McNally, S., Atchley, S., & Vazhkudai, S. S. (2018). GPU age-aware scheduling to improve the reliability of leadership jobs on titan. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 (pp. 83-93). Article 8665764 (Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SC.2018.00010

Zimmer, Christopher ; Maxwell, Don ; McNally, Stephen et al. / GPU age-aware scheduling to improve the reliability of leadership jobs on titan. Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 83-93 (Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018).

@inproceedings{5da8ac8ad279445b998bc26d1c9d7911,

title = "GPU age-aware scheduling to improve the reliability of leadership jobs on titan",

abstract = "In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ∼50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.",

author = "Christopher Zimmer and Don Maxwell and Stephen McNally and Scott Atchley and Vazhkudai, {Sudharshan S.}",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018 ; Conference date: 11-11-2018 Through 16-11-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/SC.2018.00010",

language = "English",

series = "Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "83--93",

booktitle = "Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018",

}

Zimmer, C , Maxwell, D, McNally, S, Atchley, S & Vazhkudai, SS 2018, GPU age-aware scheduling to improve the reliability of leadership jobs on titan. in Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018., 8665764, Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Institute of Electrical and Electronics Engineers Inc., pp. 83-93, 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018, Dallas, United States, 11/11/18. https://doi.org/10.1109/SC.2018.00010

GPU age-aware scheduling to improve the reliability of leadership jobs on titan. / Zimmer, Christopher ; Maxwell, Don; McNally, Stephen et al.
Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 83-93 8665764 (Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - GPU age-aware scheduling to improve the reliability of leadership jobs on titan

AU - Zimmer, Christopher

AU - Maxwell, Don

AU - McNally, Stephen

AU - Atchley, Scott

AU - Vazhkudai, Sudharshan S.

PY - 2018/7/2

Y1 - 2018/7/2

N2 - In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ∼50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.

AB - In 2015, OLCF's Titan supercomputer experienced a significant increase in GPU related job failures. The impact on jobs was serious and OLCF decided to replace ∼50% of the GPUs. Unfortunately, jobs using more than 20% of the machine (i.e., leadership jobs) continued to encounter higher levels of application failures. These jobs contained significant amounts of both the low-failure rate and high-failure rate GPUs. The impacts of these failures are more adversely felt by leadership jobs due to longer wait times, runtimes, and higher charge rates. In this work, we have designed techniques to increase the use of low-failure GPUs in leadership jobs through targeted resource allocation. We have employed two complementary techniques, updating both the system ordering and the allocation mechanisms. Using simulation, the application of these techniques resulted in a 33% increase in low-failure GPU hours being assigned to leadership jobs. Our GPU Age-Aware Scheduling has been used in production on Titan since July of 2017.

UR - http://www.scopus.com/inward/record.url?scp=85064128319&partnerID=8YFLogxK

U2 - 10.1109/SC.2018.00010

DO - 10.1109/SC.2018.00010

M3 - Conference contribution

AN - SCOPUS:85064128319

T3 - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

SP - 83

EP - 93

BT - Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018

Y2 - 11 November 2018 through 16 November 2018

ER -

Zimmer C , Maxwell D, McNally S, Atchley S, Vazhkudai SS. GPU age-aware scheduling to improve the reliability of leadership jobs on titan. In Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 83-93. 8665764. (Proceedings - International Conference for High Performance Computing, Networking, Storage, and Analysis, SC 2018). doi: 10.1109/SC.2018.00010

GPU age-aware scheduling to improve the reliability of leadership jobs on titan

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this