Elastic distributed training with fast convergence and efficient resource utilization

Guojing Cong; Guojing Cong

doi:10.1109/ICMLA52953.2021.00160

Elastic distributed training with fast convergence and efficient resource utilization

Guojing Cong, Guojing Cong

Learning Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.

Original language	English
Title of host publication	Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021
Editors	M. Arif Wani, Ishwar K. Sethi, Weisong Shi, Guangzhi Qu, Daniela Stan Raicu, Ruoming Jin
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	972-979
Number of pages	8
ISBN (Electronic)	9781665443371
DOIs	https://doi.org/10.1109/ICMLA52953.2021.00160
State	Published - 2021
Event	20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021 - Virtual, Online, United States Duration: Dec 13 2021 → Dec 16 2021

Publication series

Name	Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021

Conference

Conference	20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021
Country/Territory	United States
City	Virtual, Online
Period	12/13/21 → 12/16/21

Funding

This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. This material is based upon work supported in part by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC05-00OR22725, and in part by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC. Notice: This manuscript has been authored in part by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Funders	Funder number
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research	DE-AC05-00OR22725
Oak Ridge National Laboratory
UT-Battelle

Keywords

Cloud
Distributed Training
Elastic training
Momentum
SGD

Access to Document

10.1109/ICMLA52953.2021.00160

Cite this

Cong, G., & Cong, G. (2021). Elastic distributed training with fast convergence and efficient resource utilization. In M. A. Wani, I. K. Sethi, W. Shi, G. Qu, D. S. Raicu, & R. Jin (Eds.), Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021 (pp. 972-979). (Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA52953.2021.00160

Cong, Guojing ; Cong, Guojing. / Elastic distributed training with fast convergence and efficient resource utilization. Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021. editor / M. Arif Wani ; Ishwar K. Sethi ; Weisong Shi ; Guangzhi Qu ; Daniela Stan Raicu ; Ruoming Jin. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 972-979 (Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021).

@inproceedings{5dddd240f7ed48dfae3616002791adfd,

title = "Elastic distributed training with fast convergence and efficient resource utilization",

abstract = "Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.",

keywords = "Cloud, Distributed Training, Elastic training, Momentum, SGD",

author = "Guojing Cong and Guojing Cong",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021 ; Conference date: 13-12-2021 Through 16-12-2021",

year = "2021",

doi = "10.1109/ICMLA52953.2021.00160",

language = "English",

series = "Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "972--979",

editor = "Wani, {M. Arif} and Sethi, {Ishwar K.} and Weisong Shi and Guangzhi Qu and Raicu, {Daniela Stan} and Ruoming Jin",

booktitle = "Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021",

}

Cong, G & Cong, G 2021, Elastic distributed training with fast convergence and efficient resource utilization. in MA Wani, IK Sethi, W Shi, G Qu, DS Raicu & R Jin (eds), Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021. Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021, Institute of Electrical and Electronics Engineers Inc., pp. 972-979, 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021, Virtual, Online, United States, 12/13/21. https://doi.org/10.1109/ICMLA52953.2021.00160

Elastic distributed training with fast convergence and efficient resource utilization. / Cong, Guojing; Cong, Guojing.
Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021. ed. / M. Arif Wani; Ishwar K. Sethi; Weisong Shi; Guangzhi Qu; Daniela Stan Raicu; Ruoming Jin. Institute of Electrical and Electronics Engineers Inc., 2021. p. 972-979 (Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Elastic distributed training with fast convergence and efficient resource utilization

AU - Cong, Guojing

PY - 2021

Y1 - 2021

N2 - Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.

AB - Distributed learning is now routinely conducted on cloud as well as dedicated clusters. Training with elastic resources brings new challenges and design choices. Prior studies focus on runtime performance and assume a static algorithmic behavior. In this work, by analyzing the impact of of resource scaling on convergence, we introduce schedules for synchronous stochastic gradient descent that proactively adapt the number of learners to reduce training time and improve convergence. Our approach no longer assumes a constant number of processors throughout training. In our experiment, distributed stochastic gradient descent with dynamic schedules and reduction momentum achieves better convergence and significant speedups over prior static ones. Numerous distributed training jobs running on cloud may benefit from our approach.

KW - Cloud

KW - Distributed Training

KW - Elastic training

KW - Momentum

KW - SGD

UR - http://www.scopus.com/inward/record.url?scp=85125839779&partnerID=8YFLogxK

U2 - 10.1109/ICMLA52953.2021.00160

DO - 10.1109/ICMLA52953.2021.00160

M3 - Conference contribution

AN - SCOPUS:85125839779

T3 - Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021

SP - 972

EP - 979

BT - Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021

A2 - Wani, M. Arif

A2 - Sethi, Ishwar K.

A2 - Shi, Weisong

A2 - Qu, Guangzhi

A2 - Raicu, Daniela Stan

A2 - Jin, Ruoming

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021

Y2 - 13 December 2021 through 16 December 2021

ER -

Cong G, Cong G. Elastic distributed training with fast convergence and efficient resource utilization. In Wani MA, Sethi IK, Shi W, Qu G, Raicu DS, Jin R, editors, Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 972-979. (Proceedings - 20th IEEE International Conference on Machine Learning and Applications, ICMLA 2021). doi: 10.1109/ICMLA52953.2021.00160

Elastic distributed training with fast convergence and efficient resource utilization

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this