Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch

Elvis Rojas; Fabricio Quirós-Corella; Terry Jones; Esteban Meneses

doi:10.1007/978-3-031-04209-6_13

Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch

Elvis Rojas, Fabricio Quirós-Corella, Terry Jones, Esteban Meneses

Intelligent Systems and Facilities

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

Artificial intelligence is a transforming technology for creating new scientific discoveries, services, and products. Its full potential is achieved when massive data repositories and large-scale computing systems are available. Both factors are becoming easier to obtain daily as sensor networks constantly create open-data archives, and Moore’s law still makes supercomputing power more accessible. However, as deep learning models become larger to tackle data complexity, researchers must determine how to speed up training in those models. This paper uses an experimental approach to try to understand the algorithms and trade-offs associated with distributed deep learning. This study used the Summit supercomputer at Oak Ridge National Laboratory to determine that existing distributed deep learning mechanisms scale in execution time. However, as more nodes are used, accuracy degrades significantly. To solve this, several hyper-parameters must be tuned. The results show that optimizing those parameters is a nontrivial task. We also evaluated the impact of other scaling techniques, such as mixed precision and adaptive parameter optimization.

Original language	English
Title of host publication	High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers
Editors	Isidoro Gitler, Carlos Jaime Barrios Hernández, Esteban Meneses
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	177-192
Number of pages	16
ISBN (Print)	9783031042089
DOIs	https://doi.org/10.1007/978-3-031-04209-6_13
State	Published - 2022
Event	8th Latin American High Performance Computing Conference, CARLA 2021 - Virtual, Online Duration: Oct 6 2021 → Oct 8 2021

Publication series

Name	Communications in Computer and Information Science
Volume	1540 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	8th Latin American High Performance Computing Conference, CARLA 2021
City	Virtual, Online
Period	10/6/21 → 10/8/21

Funding

Notice: This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (https://energy.gov/downloads/doe-public-access-plan).

Funders	Funder number
U.S. Department of Energy

Keywords

Distributed deep learning
Performance
Scalability

Access to Document

10.1007/978-3-031-04209-6_13

Cite this

Rojas, E., Quirós-Corella, F., Jones, T., & Meneses, E. (2022). Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch. In I. Gitler, C. J. Barrios Hernández, & E. Meneses (Eds.), High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers (pp. 177-192). (Communications in Computer and Information Science; Vol. 1540 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-04209-6_13

Rojas, Elvis ; Quirós-Corella, Fabricio ; Jones, Terry et al. / Large-Scale Distributed Deep Learning : A Study of Mechanisms and Trade-Offs with PyTorch. High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers. editor / Isidoro Gitler ; Carlos Jaime Barrios Hernández ; Esteban Meneses. Springer Science and Business Media Deutschland GmbH, 2022. pp. 177-192 (Communications in Computer and Information Science).

@inproceedings{4593216748b441db961a63572e22d6c1,

title = "Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch",

abstract = "Artificial intelligence is a transforming technology for creating new scientific discoveries, services, and products. Its full potential is achieved when massive data repositories and large-scale computing systems are available. Both factors are becoming easier to obtain daily as sensor networks constantly create open-data archives, and Moore{\textquoteright}s law still makes supercomputing power more accessible. However, as deep learning models become larger to tackle data complexity, researchers must determine how to speed up training in those models. This paper uses an experimental approach to try to understand the algorithms and trade-offs associated with distributed deep learning. This study used the Summit supercomputer at Oak Ridge National Laboratory to determine that existing distributed deep learning mechanisms scale in execution time. However, as more nodes are used, accuracy degrades significantly. To solve this, several hyper-parameters must be tuned. The results show that optimizing those parameters is a nontrivial task. We also evaluated the impact of other scaling techniques, such as mixed precision and adaptive parameter optimization.",

keywords = "Distributed deep learning, Performance, Scalability",

author = "Elvis Rojas and Fabricio Quir{\'o}s-Corella and Terry Jones and Esteban Meneses",

note = "Publisher Copyright: {\textcopyright} 2022, UT-Battelle, LLC.; 8th Latin American High Performance Computing Conference, CARLA 2021 ; Conference date: 06-10-2021 Through 08-10-2021",

year = "2022",

doi = "10.1007/978-3-031-04209-6_13",

language = "English",

isbn = "9783031042089",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "177--192",

editor = "Isidoro Gitler and {Barrios Hern{\'a}ndez}, {Carlos Jaime} and Esteban Meneses",

booktitle = "High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers",

}

Rojas, E, Quirós-Corella, F, Jones, T & Meneses, E 2022, Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch. in I Gitler, CJ Barrios Hernández & E Meneses (eds), High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers. Communications in Computer and Information Science, vol. 1540 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 177-192, 8th Latin American High Performance Computing Conference, CARLA 2021, Virtual, Online, 10/6/21. https://doi.org/10.1007/978-3-031-04209-6_13

Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch. / Rojas, Elvis; Quirós-Corella, Fabricio; Jones, Terry et al.
High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers. ed. / Isidoro Gitler; Carlos Jaime Barrios Hernández; Esteban Meneses. Springer Science and Business Media Deutschland GmbH, 2022. p. 177-192 (Communications in Computer and Information Science; Vol. 1540 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Large-Scale Distributed Deep Learning

T2 - 8th Latin American High Performance Computing Conference, CARLA 2021

AU - Rojas, Elvis

AU - Quirós-Corella, Fabricio

AU - Jones, Terry

AU - Meneses, Esteban

PY - 2022

Y1 - 2022

N2 - Artificial intelligence is a transforming technology for creating new scientific discoveries, services, and products. Its full potential is achieved when massive data repositories and large-scale computing systems are available. Both factors are becoming easier to obtain daily as sensor networks constantly create open-data archives, and Moore’s law still makes supercomputing power more accessible. However, as deep learning models become larger to tackle data complexity, researchers must determine how to speed up training in those models. This paper uses an experimental approach to try to understand the algorithms and trade-offs associated with distributed deep learning. This study used the Summit supercomputer at Oak Ridge National Laboratory to determine that existing distributed deep learning mechanisms scale in execution time. However, as more nodes are used, accuracy degrades significantly. To solve this, several hyper-parameters must be tuned. The results show that optimizing those parameters is a nontrivial task. We also evaluated the impact of other scaling techniques, such as mixed precision and adaptive parameter optimization.

AB - Artificial intelligence is a transforming technology for creating new scientific discoveries, services, and products. Its full potential is achieved when massive data repositories and large-scale computing systems are available. Both factors are becoming easier to obtain daily as sensor networks constantly create open-data archives, and Moore’s law still makes supercomputing power more accessible. However, as deep learning models become larger to tackle data complexity, researchers must determine how to speed up training in those models. This paper uses an experimental approach to try to understand the algorithms and trade-offs associated with distributed deep learning. This study used the Summit supercomputer at Oak Ridge National Laboratory to determine that existing distributed deep learning mechanisms scale in execution time. However, as more nodes are used, accuracy degrades significantly. To solve this, several hyper-parameters must be tuned. The results show that optimizing those parameters is a nontrivial task. We also evaluated the impact of other scaling techniques, such as mixed precision and adaptive parameter optimization.

KW - Distributed deep learning

KW - Performance

KW - Scalability

UR - http://www.scopus.com/inward/record.url?scp=85128986740&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-04209-6_13

DO - 10.1007/978-3-031-04209-6_13

M3 - Conference contribution

AN - SCOPUS:85128986740

SN - 9783031042089

T3 - Communications in Computer and Information Science

SP - 177

EP - 192

BT - High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers

A2 - Gitler, Isidoro

A2 - Barrios Hernández, Carlos Jaime

A2 - Meneses, Esteban

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 6 October 2021 through 8 October 2021

ER -

Rojas E, Quirós-Corella F, Jones T, Meneses E. Large-Scale Distributed Deep Learning: A Study of Mechanisms and Trade-Offs with PyTorch. In Gitler I, Barrios Hernández CJ, Meneses E, editors, High Performance Computing - 8th Latin American Conference, CARLA 2021, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 177-192. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-04209-6_13