Optimizing deep learning hyper-parameters through an evolutionary algorithm

Steven R. Young; Derek C. Rose; Thomas P. Karnowski; Seung Hwan Lim; Robert M. Patton

doi:10.1145/2834892.2834896

Optimizing deep learning hyper-parameters through an evolutionary algorithm

Steven R. Young, Derek C. Rose, Thomas P. Karnowski, Seung Hwan Lim, Robert M. Patton

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

338 Scopus citations

Abstract

There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.

Original language	English
Title of host publication	Proceedings of MLHPC 2015
Subtitle of host publication	Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9781450340069
DOIs	https://doi.org/10.1145/2834892.2834896
State	Published - Nov 15 2015
Event	Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015 - Austin, United States Duration: Nov 15 2015 → …

Publication series

Name	Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015
Country/Territory	United States
City	Austin
Period	11/15/15 → …

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan(http://energy.gov/downloads/doe-public-access-plan). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Keywords

Convolutional neural networks
Deep learning
Evolutionary algorithm
Hyper-parameter optimization

Access to Document

10.1145/2834892.2834896

Cite this

Young, S. R., Rose, D. C., Karnowski, T. P., Lim, S. H., & Patton, R. M. (2015). Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis Article a4 (Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/2834892.2834896

Young, Steven R. ; Rose, Derek C. ; Karnowski, Thomas P. et al. / Optimizing deep learning hyper-parameters through an evolutionary algorithm. Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2015. (Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{a31aec620fc746eaaaae4ef84f8b3fbd,

title = "Optimizing deep learning hyper-parameters through an evolutionary algorithm",

abstract = "There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.",

keywords = "Convolutional neural networks, Deep learning, Evolutionary algorithm, Hyper-parameter optimization",

author = "Young, {Steven R.} and Rose, {Derek C.} and Karnowski, {Thomas P.} and Lim, {Seung Hwan} and Patton, {Robert M.}",

note = "Publisher Copyright: {\textcopyright} 2015 ACM.; Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015 ; Conference date: 15-11-2015",

year = "2015",

month = nov,

day = "15",

doi = "10.1145/2834892.2834896",

language = "English",

series = "Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Association for Computing Machinery, Inc",

booktitle = "Proceedings of MLHPC 2015",

}

Young, SR , Rose, DC , Karnowski, TP , Lim, SH & Patton, RM 2015, Optimizing deep learning hyper-parameters through an evolutionary algorithm. in Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis., a4, Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis, Association for Computing Machinery, Inc, Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015, Austin, United States, 11/15/15. https://doi.org/10.1145/2834892.2834896

Optimizing deep learning hyper-parameters through an evolutionary algorithm. / Young, Steven R.; Rose, Derek C.; Karnowski, Thomas P. et al.
Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2015. a4 (Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Optimizing deep learning hyper-parameters through an evolutionary algorithm

AU - Young, Steven R.

AU - Rose, Derek C.

AU - Karnowski, Thomas P.

AU - Lim, Seung Hwan

AU - Patton, Robert M.

PY - 2015/11/15

Y1 - 2015/11/15

N2 - There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.

AB - There has been a recent surge of success in utilizing Deep Learning (DL) in imaging and speech applications for its relatively automatic feature generation and, in particular for convolutional neural networks (CNNs), high accuracy classification abilities. While these models learn their parameters through data-driven methods, model selection (as architecture construction) through hyper-parameter choices remains a tedious and highly intuition driven task. To address this, Multi-node Evolutionary Neural Networks for Deep Learning (MENNDL) is proposed as a method for automating network selection on computational clusters through hyper-parameter optimization performed via genetic algorithms.

KW - Convolutional neural networks

KW - Deep learning

KW - Evolutionary algorithm

KW - Hyper-parameter optimization

UR - http://www.scopus.com/inward/record.url?scp=84959225396&partnerID=8YFLogxK

U2 - 10.1145/2834892.2834896

DO - 10.1145/2834892.2834896

M3 - Conference contribution

AN - SCOPUS:84959225396

T3 - Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis

BT - Proceedings of MLHPC 2015

PB - Association for Computing Machinery, Inc

T2 - Workshop on Machine Learning in High-Performance Computing Environments, MLHPC 2015

Y2 - 15 November 2015

ER -

Young SR , Rose DC , Karnowski TP , Lim SH , Patton RM. Optimizing deep learning hyper-parameters through an evolutionary algorithm. In Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc. 2015. a4. (Proceedings of MLHPC 2015: Machine Learning in High-Performance Computing Environments - Held in conjunction with SC 2015: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1145/2834892.2834896

Optimizing deep learning hyper-parameters through an evolutionary algorithm

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this