Distributed Bayesian optimization of deep reinforcement learning algorithms

M. Todd Young; Jacob D. Hinkle; Ramakrishnan Kannan; Arvind Ramanathan

doi:10.1016/j.jpdc.2019.07.008

Distributed Bayesian optimization of deep reinforcement learning algorithms

M. Todd Young, Jacob D. Hinkle, Ramakrishnan Kannan, Arvind Ramanathan

Discrete Algorithms

Research output: Contribution to journal › Article › peer-review

28 Scopus citations

Abstract

Significant strides have been made in supervised learning settings thanks to the successful application of deep learning. Now, recent work has brought the techniques of deep learning to bear on sequential decision processes in the area of deep reinforcement learning (DRL). Currently, little is known regarding hyperparameter optimization for DRL algorithms. Given that DRL algorithms are computationally intensive to train, and are known to be sample inefficient, optimizing model hyperparameters for DRL presents significant challenges to established techniques. We provide an open source, distributed Bayesian model-based optimization algorithm, HyperSpace, and show that it consistently outperforms standard hyperparameter optimization techniques across three DRL algorithms.

Original language	English
Pages (from-to)	43-52
Number of pages	10
Journal	Journal of Parallel and Distributed Computing
Volume	139
DOIs	https://doi.org/10.1016/j.jpdc.2019.07.008
State	Published - May 2020

Funding

This work has been supported in part by the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) program established by the U.S. Department of Energy (DOE) and the National Cancer Institute (NCI) of the National Institutes of Health. This work was performed under the auspices of the U.S. Department of Energy by Argonne National Laboratory under Contract DE-AC02-06-CH11357, Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344, Los Alamos National Laboratory under Contract DE-AC5206NA25396, and Oak Ridge National Laboratory under Contract DE-AC05-00OR22725. This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This work has been supported in part by the Joint Design of Advanced Computing Solutions for Cancer (JDACS4C) program established by the U.S. Department of Energy (DOE) and the National Cancer Institute (NCI) of the National Institutes of Health. This work was performed under the auspices of the U.S. Department of Energy by Argonne National Laboratory under Contract DE-AC02-06-CH11357 , Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 , Los Alamos National Laboratory under Contract DE-AC5206NA25396 , and Oak Ridge National Laboratory under Contract DE-AC05-00OR22725 . This research was supported by the Exascale Computing Project ( 17-SC-20-SC ), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory.

Funders	Funder number
DOE Public Access Plan
U.S. Department of Energy Office of Science
United States Government
National Institutes of Health
U.S. Department of Energy
National Cancer Institute
National Nuclear Security Administration
Argonne National Laboratory	DE-AC02-06-CH11357
Lawrence Livermore National Laboratory	DE-AC52-07NA27344
Oak Ridge National Laboratory	17-SC-20-SC, DE-AC05-00OR22725
Los Alamos National Laboratory	DE-AC5206NA25396

Keywords

Bayesian optimization
Deep reinforcement learning

Access to Document

10.1016/j.jpdc.2019.07.008

Cite this

@article{9ea706eb7ec34645a3003270e3d6b920,

title = "Distributed Bayesian optimization of deep reinforcement learning algorithms",

abstract = "Significant strides have been made in supervised learning settings thanks to the successful application of deep learning. Now, recent work has brought the techniques of deep learning to bear on sequential decision processes in the area of deep reinforcement learning (DRL). Currently, little is known regarding hyperparameter optimization for DRL algorithms. Given that DRL algorithms are computationally intensive to train, and are known to be sample inefficient, optimizing model hyperparameters for DRL presents significant challenges to established techniques. We provide an open source, distributed Bayesian model-based optimization algorithm, HyperSpace, and show that it consistently outperforms standard hyperparameter optimization techniques across three DRL algorithms.",

keywords = "Bayesian optimization, Deep reinforcement learning",

author = "Young, {M. Todd} and Hinkle, {Jacob D.} and Ramakrishnan Kannan and Arvind Ramanathan",

note = "Publisher Copyright: {\textcopyright} 2019 The Author",

year = "2020",

month = may,

doi = "10.1016/j.jpdc.2019.07.008",

language = "English",

volume = "139",

pages = "43--52",

journal = "Journal of Parallel and Distributed Computing",

issn = "0743-7315",

publisher = "Elsevier",

}

TY - JOUR

T1 - Distributed Bayesian optimization of deep reinforcement learning algorithms

AU - Young, M. Todd

AU - Hinkle, Jacob D.

AU - Kannan, Ramakrishnan

AU - Ramanathan, Arvind

PY - 2020/5

Y1 - 2020/5

N2 - Significant strides have been made in supervised learning settings thanks to the successful application of deep learning. Now, recent work has brought the techniques of deep learning to bear on sequential decision processes in the area of deep reinforcement learning (DRL). Currently, little is known regarding hyperparameter optimization for DRL algorithms. Given that DRL algorithms are computationally intensive to train, and are known to be sample inefficient, optimizing model hyperparameters for DRL presents significant challenges to established techniques. We provide an open source, distributed Bayesian model-based optimization algorithm, HyperSpace, and show that it consistently outperforms standard hyperparameter optimization techniques across three DRL algorithms.

AB - Significant strides have been made in supervised learning settings thanks to the successful application of deep learning. Now, recent work has brought the techniques of deep learning to bear on sequential decision processes in the area of deep reinforcement learning (DRL). Currently, little is known regarding hyperparameter optimization for DRL algorithms. Given that DRL algorithms are computationally intensive to train, and are known to be sample inefficient, optimizing model hyperparameters for DRL presents significant challenges to established techniques. We provide an open source, distributed Bayesian model-based optimization algorithm, HyperSpace, and show that it consistently outperforms standard hyperparameter optimization techniques across three DRL algorithms.

KW - Bayesian optimization

KW - Deep reinforcement learning

UR - http://www.scopus.com/inward/record.url?scp=85079395340&partnerID=8YFLogxK

U2 - 10.1016/j.jpdc.2019.07.008

DO - 10.1016/j.jpdc.2019.07.008

M3 - Article

AN - SCOPUS:85079395340

SN - 0743-7315

VL - 139

SP - 43

EP - 52

JO - Journal of Parallel and Distributed Computing

JF - Journal of Parallel and Distributed Computing

ER -

Distributed Bayesian optimization of deep reinforcement learning algorithms

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this