A scalable algorithm for the optimization of neural network architectures

Massimiliano Lupo Pasini; Junqi Yin; Ying Wai Li; Markus Eisenbach

doi:10.1016/j.parco.2021.102788

A scalable algorithm for the optimization of neural network architectures

Massimiliano Lupo Pasini, Junqi Yin, Ying Wai Li, Markus Eisenbach

Research output: Contribution to journal › Article › peer-review

5 Scopus citations

Abstract

We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

Original language	English
Article number	102788
Journal	Parallel Computing
Volume	104-105
DOIs	https://doi.org/10.1016/j.parco.2021.102788
State	Published - Jul 2021

Funding

This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).Massimiliano Lupo Pasini thanks Dr. Vladimir Protopopescu for his valuable feedback in the preparation of this manuscript and three anonymous reviewers for their very useful comments and suggestions. This work is supported in part by the Office of Science of the US Department of Energy (DOE) and by the LDRD Program of Oak Ridge National Laboratory, USA. This work used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. Y. W. Li was partly supported by the LDRD Program of Los Alamos National Laboratory (LANL), USA under project number 20190005DR. LANL is operated by Triad National Security, LLC, for the National Nuclear Security Administration of U.S. Department of Energy (Contract No. 89233218CNA000001). This document number is LA-UR-21-20936. This work is supported in part by the Office of Science of the US Department of Energy (DOE) and by the LDRD Program of Oak Ridge National Laboratory, USA . This work used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725. Y. W. Li was partly supported by the LDRD Program of Los Alamos National Laboratory (LANL), USA under project number 20190005DR . LANL is operated by Triad National Security, LLC, for the National Nuclear Security Administration of U.S. Department of Energy (Contract No. 89233218CNA000001). This document number is LA-UR-21-20936.

Funders	Funder number
DOE Public Access Plan
U.S. Department of Energy	89233218CNA000001, LA-UR-21-20936
Office of Science	DE-AC05-00OR22725
National Nuclear Security Administration
Oak Ridge National Laboratory
Laboratory Directed Research and Development
Los Alamos National Laboratory	20190005DR

Keywords

Adaptive algorithms
Deep learning
Greedy constructive algorithms
Hyperparameter optimization
Neural network architecture
Random search

Access to Document

10.1016/j.parco.2021.102788

Cite this

@article{efb3f6336f8a4331888b4372c78312d6,

title = "A scalable algorithm for the optimization of neural network architectures",

abstract = "We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.",

keywords = "Adaptive algorithms, Deep learning, Greedy constructive algorithms, Hyperparameter optimization, Neural network architecture, Random search",

author = "{Lupo Pasini}, Massimiliano and Junqi Yin and Li, {Ying Wai} and Markus Eisenbach",

note = "Publisher Copyright: {\textcopyright} 2021 Elsevier B.V.",

year = "2021",

month = jul,

doi = "10.1016/j.parco.2021.102788",

language = "English",

volume = "104-105",

journal = "Parallel Computing",

issn = "0167-8191",

publisher = "Elsevier",

}

TY - JOUR

T1 - A scalable algorithm for the optimization of neural network architectures

AU - Lupo Pasini, Massimiliano

AU - Yin, Junqi

AU - Li, Ying Wai

AU - Eisenbach, Markus

PY - 2021/7

Y1 - 2021/7

N2 - We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

AB - We propose a new scalable method to optimize the architecture of an artificial neural network. The proposed algorithm, called Greedy Search for Neural Network Architecture, aims to determine a neural network with minimal number of layers that is at least as performant as neural networks of the same structure identified by other hyperparameter search algorithms in terms of accuracy and computational cost. Numerical results performed on benchmark datasets show that, for these datasets, our method outperforms state-of-the-art hyperparameter optimization algorithms in terms of attainable predictive performance by the selected neural network architecture, and time-to-solution for the hyperparameter optimization to complete.

KW - Adaptive algorithms

KW - Deep learning

KW - Greedy constructive algorithms

KW - Hyperparameter optimization

KW - Neural network architecture

KW - Random search

UR - http://www.scopus.com/inward/record.url?scp=85107104202&partnerID=8YFLogxK

U2 - 10.1016/j.parco.2021.102788

DO - 10.1016/j.parco.2021.102788

M3 - Article

AN - SCOPUS:85107104202

SN - 0167-8191

VL - 104-105

JO - Parallel Computing

JF - Parallel Computing

M1 - 102788

ER -

A scalable algorithm for the optimization of neural network architectures

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this