TY - GEN
T1 - Accelerate distributed stochastic descent for nonconvex optimization with momentum
AU - Cong, Guojing
AU - Liu, Tianyi
N1 - Publisher Copyright:
© 2020 IEEE.
PY - 2020/11
Y1 - 2020/11
N2 - Momentum method has been used extensively in optimizers for deep learning. Recent studies show that distributed training through K-step averaging has many nice properties. We propose a momentum method for such model averaging approaches. At each individual learner level traditional stochastic gradient is applied. At the meta-level (global learner level), one momentum term is applied and we call it block momentum. We analyze the convergence and scaling properties of such momentum methods. Our experimental results show that block momentum not only accelerates training, but also achieves better results.
AB - Momentum method has been used extensively in optimizers for deep learning. Recent studies show that distributed training through K-step averaging has many nice properties. We propose a momentum method for such model averaging approaches. At each individual learner level traditional stochastic gradient is applied. At the meta-level (global learner level), one momentum term is applied and we call it block momentum. We analyze the convergence and scaling properties of such momentum methods. Our experimental results show that block momentum not only accelerates training, but also achieves better results.
UR - http://www.scopus.com/inward/record.url?scp=85105403348&partnerID=8YFLogxK
U2 - 10.1109/MLHPCAI4S51975.2020.00011
DO - 10.1109/MLHPCAI4S51975.2020.00011
M3 - Conference contribution
AN - SCOPUS:85105403348
T3 - Proceedings of 2020 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments, MLHPC 2020 and Workshop on Artificial Intelligence and Machine Learning for Scientific Applications, AI4S 2020 - Held in conjunction with SC 2020: The International Conference for High Performance Computing, Networking, Storage and Analysis
SP - 29
EP - 39
BT - Proceedings of 2020 IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments, MLHPC 2020 and Workshop on Artificial Intelligence and Machine Learning for Scientific Applications, AI4S 2020 - Held in conjunction with SC 2020
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 6th IEEE/ACM Workshop on Machine Learning in High Performance Computing Environments, MLHPC 2020 and 1st Workshop on Artificial Intelligence and Machine Learning for Scientific Applications, AI4S 2020
Y2 - 12 November 2020
ER -