Strategies to deploy and scale deep learning on the summit supercomputer

Junqi Yin; Shubhankar Gahlot; Nouamane Laanait; Ketan Maheshwari; Jack Morrison; Sajal Dash; Mallikarjun Shankar

doi:10.1109/DLS49591.2019.00016

Strategies to deploy and scale deep learning on the summit supercomputer

Junqi Yin
, Shubhankar Gahlot
, Nouamane Laanait
, Ketan Maheshwari
, Jack Morrison
, Sajal Dash
, Mallikarjun Shankar

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

20 Scopus citations

Abstract

The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).

Original language	English
Title of host publication	Proceedings of DLS 2019
Subtitle of host publication	Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	84-94
Number of pages	11
ISBN (Electronic)	9781728160115
DOIs	https://doi.org/10.1109/DLS49591.2019.00016
State	Published - Nov 2019
Event	3rd IEEE/ACM Workshop on Deep Learning on Supercomputers, DLS 2019 - Denver, United States Duration: Nov 17 2019 → …

Publication series

Name	Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	3rd IEEE/ACM Workshop on Deep Learning on Supercomputers, DLS 2019
Country/Territory	United States
City	Denver
Period	11/17/19 → …

Funding

This research was sponsored by and used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility and the Compute and Data Environment for Science (CADES) at the Oak Ridge National Laboratory supported by the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Part of the computer time was also provided by the INCITE program.

Keywords

HPC
Performance evaluation
Scalable machine learning
Software deployment

Access to Document

10.1109/DLS49591.2019.00016

Cite this

Yin, J., Gahlot, S., Laanait, N., Maheshwari, K., Morrison, J., Dash, S., & Shankar, M. (2019). Strategies to deploy and scale deep learning on the summit supercomputer. In Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 84-94). Article 8945109 (Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DLS49591.2019.00016

Yin, Junqi ; Gahlot, Shubhankar ; Laanait, Nouamane et al. / Strategies to deploy and scale deep learning on the summit supercomputer. Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 84-94 (Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{2a6700ae314346a8a6f02a4447244166,

title = "Strategies to deploy and scale deep learning on the summit supercomputer",

abstract = "The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87\% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).",

keywords = "HPC, Performance evaluation, Scalable machine learning, Software deployment",

author = "Junqi Yin and Shubhankar Gahlot and Nouamane Laanait and Ketan Maheshwari and Jack Morrison and Sajal Dash and Mallikarjun Shankar",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 3rd IEEE/ACM Workshop on Deep Learning on Supercomputers, DLS 2019 ; Conference date: 17-11-2019",

year = "2019",

month = nov,

doi = "10.1109/DLS49591.2019.00016",

language = "English",

series = "Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "84--94",

booktitle = "Proceedings of DLS 2019",

}

Yin, J, Gahlot, S, Laanait, N, Maheshwari, K, Morrison, J, Dash, S & Shankar, M 2019, Strategies to deploy and scale deep learning on the summit supercomputer. in Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis., 8945109, Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 84-94, 3rd IEEE/ACM Workshop on Deep Learning on Supercomputers, DLS 2019, Denver, United States, 11/17/19. https://doi.org/10.1109/DLS49591.2019.00016

Strategies to deploy and scale deep learning on the summit supercomputer. / Yin, Junqi; Gahlot, Shubhankar; Laanait, Nouamane et al.
Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2019. p. 84-94 8945109 (Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Strategies to deploy and scale deep learning on the summit supercomputer

AU - Yin, Junqi

AU - Gahlot, Shubhankar

AU - Laanait, Nouamane

AU - Maheshwari, Ketan

AU - Morrison, Jack

AU - Dash, Sajal

AU - Shankar, Mallikarjun

PY - 2019/11

Y1 - 2019/11

N2 - The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).

AB - The rapid growth and wide applicability of Deep Learning (DL) frameworks poses challenges to computing centers which need to deploy and support the software, and also to domain scientists who have to keep up with the system environment and scale up scientific exploration through DL. We offer recommendations for deploying and scaling DL frameworks on the Summit supercomputer, currently atop the Top500 list, at the Oak Ridge National Laboratory Leadership Computing Facility (OLCF). We discuss DL software deployment in the form of containers, and compare performance of native-built frameworks and containerized deployment. Software containers show no noticeable negative performance impact and exhibit faster Python loading times and promise easier maintenance. To explore strategies for scaling up DL model training campaigns, we assess DL compute kernel performance, discuss and recommend I/O data formats and staging, and identify communication needs for scalable message exchange for DL runs at scale. We recommend that users take a step-wise tuning approach beginning with algorithmic kernel choice, node I/O configuration, and communications tuning as best-practice. We present baseline examples of scaling efficiency 87% for a DL run of ResNet50 running on 1024 nodes (6144 V100 GPUs).

KW - HPC

KW - Performance evaluation

KW - Scalable machine learning

KW - Software deployment

UR - https://www.scopus.com/pages/publications/85078093675

U2 - 10.1109/DLS49591.2019.00016

DO - 10.1109/DLS49591.2019.00016

M3 - Conference contribution

AN - SCOPUS:85078093675

T3 - Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 84

EP - 94

BT - Proceedings of DLS 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 3rd IEEE/ACM Workshop on Deep Learning on Supercomputers, DLS 2019

Y2 - 17 November 2019

ER -

Yin J, Gahlot S, Laanait N, Maheshwari K, Morrison J, Dash S et al. Strategies to deploy and scale deep learning on the summit supercomputer. In Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2019. p. 84-94. 8945109. (Proceedings of DLS 2019: Deep Learning on Supercomputers - Held in conjunction with SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/DLS49591.2019.00016

Strategies to deploy and scale deep learning on the summit supercomputer

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this