Towards native execution of deep learning on a leadership-class HPC system

Srikanth Yoginath; Maksudul Alam; Arvind Ramanathan; Debsindhu Bhowmik; Nouamane Laanait; Kalyan S. Perumalla

doi:10.1109/IPDPSW.2019.00160

Towards native execution of deep learning on a leadership-class HPC system

Srikanth Yoginath, Maksudul Alam, Arvind Ramanathan, Debsindhu Bhowmik, Nouamane Laanait, Kalyan S. Perumalla

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Large parallel machines generally offer the best parallel performance with 'native execution' that is achieved using codes developed with the optimized compilers, communication libraries, and runtimes offered on the machines. In this paper, we report and analyze performance results from native execution of deep learning on a leadership-class high-performance computing (HPC) system. Using our new code called DeepEx, we present a study of the parallel speed up and convergence rates of learning achieved with native parallel execution. In the trade-off between computational parallelism and synchronized convergence, we first focus on maximizing parallelism while still obtaining convergence. Scaling results are reported from execution on up to 15,000 GPUs using two scientific data sets from atom microscopy and protein folding applications, and also using the popular ImageNet data set. In terms of the traditional measure of parallel speed up, excellent scaling is observed up to 12,000 GPUs. Additionally, accounting for convergence rates of deep learning accuracy or error, a deep learning-specific metric called 'learning speed up' is also tracked. The performance results indicate the need to evaluate parallel deep learning execution in terms of learning speed up, and point to additional directions for improved exploitation of high-end HPC systems.

Original language	English
Title of host publication	Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	941-950
Number of pages	10
ISBN (Electronic)	9781728135106
DOIs	https://doi.org/10.1109/IPDPSW.2019.00160
State	Published - May 2019
Event	33rd IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019 - Rio de Janeiro, Brazil Duration: May 20 2019 → May 24 2019

Publication series

Name	Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019

Conference

Conference	33rd IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019
Country/Territory	Brazil
City	Rio de Janeiro
Period	05/20/19 → 05/24/19

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

Keywords

Deep Learning
Learning Speedup
Massively Parallel Systems
Parallel Speedup

Access to Document

10.1109/IPDPSW.2019.00160

Cite this

Yoginath, S., Alam, M., Ramanathan, A., Bhowmik, D., Laanait, N., & Perumalla, K. S. (2019). Towards native execution of deep learning on a leadership-class HPC system. In Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019 (pp. 941-950). Article 8778212 (Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPSW.2019.00160

Yoginath, Srikanth ; Alam, Maksudul ; Ramanathan, Arvind et al. / Towards native execution of deep learning on a leadership-class HPC system. Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 941-950 (Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019).

@inproceedings{76788b11d3a445d88e1ee5ccbfebeab1,

title = "Towards native execution of deep learning on a leadership-class HPC system",

abstract = "Large parallel machines generally offer the best parallel performance with 'native execution' that is achieved using codes developed with the optimized compilers, communication libraries, and runtimes offered on the machines. In this paper, we report and analyze performance results from native execution of deep learning on a leadership-class high-performance computing (HPC) system. Using our new code called DeepEx, we present a study of the parallel speed up and convergence rates of learning achieved with native parallel execution. In the trade-off between computational parallelism and synchronized convergence, we first focus on maximizing parallelism while still obtaining convergence. Scaling results are reported from execution on up to 15,000 GPUs using two scientific data sets from atom microscopy and protein folding applications, and also using the popular ImageNet data set. In terms of the traditional measure of parallel speed up, excellent scaling is observed up to 12,000 GPUs. Additionally, accounting for convergence rates of deep learning accuracy or error, a deep learning-specific metric called 'learning speed up' is also tracked. The performance results indicate the need to evaluate parallel deep learning execution in terms of learning speed up, and point to additional directions for improved exploitation of high-end HPC systems.",

keywords = "Deep Learning, Learning Speedup, Massively Parallel Systems, Parallel Speedup",

author = "Srikanth Yoginath and Maksudul Alam and Arvind Ramanathan and Debsindhu Bhowmik and Nouamane Laanait and Perumalla, {Kalyan S.}",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 33rd IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019 ; Conference date: 20-05-2019 Through 24-05-2019",

year = "2019",

month = may,

doi = "10.1109/IPDPSW.2019.00160",

language = "English",

series = "Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "941--950",

booktitle = "Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019",

}

Yoginath, S , Alam, M, Ramanathan, A, Bhowmik, D, Laanait, N & Perumalla, KS 2019, Towards native execution of deep learning on a leadership-class HPC system. in Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019., 8778212, Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019, Institute of Electrical and Electronics Engineers Inc., pp. 941-950, 33rd IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019, Rio de Janeiro, Brazil, 05/20/19. https://doi.org/10.1109/IPDPSW.2019.00160

Towards native execution of deep learning on a leadership-class HPC system. / Yoginath, Srikanth ; Alam, Maksudul; Ramanathan, Arvind et al.
Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 941-950 8778212 (Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Towards native execution of deep learning on a leadership-class HPC system

AU - Yoginath, Srikanth

AU - Alam, Maksudul

AU - Ramanathan, Arvind

AU - Bhowmik, Debsindhu

AU - Laanait, Nouamane

AU - Perumalla, Kalyan S.

PY - 2019/5

Y1 - 2019/5

N2 - Large parallel machines generally offer the best parallel performance with 'native execution' that is achieved using codes developed with the optimized compilers, communication libraries, and runtimes offered on the machines. In this paper, we report and analyze performance results from native execution of deep learning on a leadership-class high-performance computing (HPC) system. Using our new code called DeepEx, we present a study of the parallel speed up and convergence rates of learning achieved with native parallel execution. In the trade-off between computational parallelism and synchronized convergence, we first focus on maximizing parallelism while still obtaining convergence. Scaling results are reported from execution on up to 15,000 GPUs using two scientific data sets from atom microscopy and protein folding applications, and also using the popular ImageNet data set. In terms of the traditional measure of parallel speed up, excellent scaling is observed up to 12,000 GPUs. Additionally, accounting for convergence rates of deep learning accuracy or error, a deep learning-specific metric called 'learning speed up' is also tracked. The performance results indicate the need to evaluate parallel deep learning execution in terms of learning speed up, and point to additional directions for improved exploitation of high-end HPC systems.

AB - Large parallel machines generally offer the best parallel performance with 'native execution' that is achieved using codes developed with the optimized compilers, communication libraries, and runtimes offered on the machines. In this paper, we report and analyze performance results from native execution of deep learning on a leadership-class high-performance computing (HPC) system. Using our new code called DeepEx, we present a study of the parallel speed up and convergence rates of learning achieved with native parallel execution. In the trade-off between computational parallelism and synchronized convergence, we first focus on maximizing parallelism while still obtaining convergence. Scaling results are reported from execution on up to 15,000 GPUs using two scientific data sets from atom microscopy and protein folding applications, and also using the popular ImageNet data set. In terms of the traditional measure of parallel speed up, excellent scaling is observed up to 12,000 GPUs. Additionally, accounting for convergence rates of deep learning accuracy or error, a deep learning-specific metric called 'learning speed up' is also tracked. The performance results indicate the need to evaluate parallel deep learning execution in terms of learning speed up, and point to additional directions for improved exploitation of high-end HPC systems.

KW - Deep Learning

KW - Learning Speedup

KW - Massively Parallel Systems

KW - Parallel Speedup

UR - http://www.scopus.com/inward/record.url?scp=85070383771&partnerID=8YFLogxK

U2 - 10.1109/IPDPSW.2019.00160

DO - 10.1109/IPDPSW.2019.00160

M3 - Conference contribution

AN - SCOPUS:85070383771

T3 - Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019

SP - 941

EP - 950

BT - Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 33rd IEEE International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019

Y2 - 20 May 2019 through 24 May 2019

ER -

Yoginath S , Alam M, Ramanathan A, Bhowmik D, Laanait N, Perumalla KS. Towards native execution of deep learning on a leadership-class HPC system. In Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 941-950. 8778212. (Proceedings - 2019 IEEE 33rd International Parallel and Distributed Processing Symposium Workshops, IPDPSW 2019). doi: 10.1109/IPDPSW.2019.00160

Towards native execution of deep learning on a leadership-class HPC system

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this