Ensembles of Networks Produced from Neural Architecture Search

Emily J. Herron; Steven R. Young; Thomas E. Potok

doi:10.1007/978-3-030-59851-8_14

Ensembles of Networks Produced from Neural Architecture Search

Emily J. Herron, Steven R. Young, Thomas E. Potok

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Neural architecture search (NAS) is a popular topic at the intersection of deep learning and high performance computing. NAS focuses on optimizing the architecture of neural networks along with their hyperparameters in order to produce networks with superior performance. Much of the focus has been on how to produce a single best network to solve a machine learning problem, but as NAS methods produce many networks that work very well, this affords the opportunity to ensemble these networks to produce an improved result. Additionally, the diversity of network structures produced by NAS drives a natural bias towards diversity of predictions produced by the individual networks. This results in an improved ensemble over simply creating an ensemble that contains duplicates of the best network architecture retrained to have unique weights.

Original language	English
Title of host publication	High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers
Editors	Heike Jagode, Hartwig Anzt, Guido Juckeland, Hatem Ltaief
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	223-234
Number of pages	12
ISBN (Print)	9783030598501
DOIs	https://doi.org/10.1007/978-3-030-59851-8_14
State	Published - 2020
Event	35th International Conference on High Performance Computing , ISC High Performance 2020 - Frankfurt am Main, Germany Duration: Jun 21 2020 → Jun 25 2020

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	12321 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	35th International Conference on High Performance Computing , ISC High Performance 2020
Country/Territory	Germany
City	Frankfurt am Main
Period	06/21/20 → 06/25/20

Funding

This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Robinson Pino, program manager, under contract number DE-AC05-00OR22725. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

Funders	Funder number
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research	DE-AC05-00OR22725

Keywords

Ensembles
High performance computing
Neural architecture search

Access to Document

10.1007/978-3-030-59851-8_14

Cite this

Herron, E. J., Young, S. R., & Potok, T. E. (2020). Ensembles of Networks Produced from Neural Architecture Search. In H. Jagode, H. Anzt, G. Juckeland, & H. Ltaief (Eds.), High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers (pp. 223-234). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12321 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-59851-8_14

Herron, Emily J. ; Young, Steven R. ; Potok, Thomas E. / Ensembles of Networks Produced from Neural Architecture Search. High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers. editor / Heike Jagode ; Hartwig Anzt ; Guido Juckeland ; Hatem Ltaief. Springer Science and Business Media Deutschland GmbH, 2020. pp. 223-234 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{eebc12127b6c4da6af022eae500193db,

title = "Ensembles of Networks Produced from Neural Architecture Search",

abstract = "Neural architecture search (NAS) is a popular topic at the intersection of deep learning and high performance computing. NAS focuses on optimizing the architecture of neural networks along with their hyperparameters in order to produce networks with superior performance. Much of the focus has been on how to produce a single best network to solve a machine learning problem, but as NAS methods produce many networks that work very well, this affords the opportunity to ensemble these networks to produce an improved result. Additionally, the diversity of network structures produced by NAS drives a natural bias towards diversity of predictions produced by the individual networks. This results in an improved ensemble over simply creating an ensemble that contains duplicates of the best network architecture retrained to have unique weights.",

keywords = "Ensembles, High performance computing, Neural architecture search",

author = "Herron, {Emily J.} and Young, {Steven R.} and Potok, {Thomas E.}",

note = "Publisher Copyright: {\textcopyright} 2020, Springer Nature Switzerland AG.; 35th International Conference on High Performance Computing , ISC High Performance 2020 ; Conference date: 21-06-2020 Through 25-06-2020",

year = "2020",

doi = "10.1007/978-3-030-59851-8_14",

language = "English",

isbn = "9783030598501",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "223--234",

editor = "Heike Jagode and Hartwig Anzt and Guido Juckeland and Hatem Ltaief",

booktitle = "High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers",

}

Herron, EJ , Young, SR & Potok, TE 2020, Ensembles of Networks Produced from Neural Architecture Search. in H Jagode, H Anzt, G Juckeland & H Ltaief (eds), High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 12321 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 223-234, 35th International Conference on High Performance Computing , ISC High Performance 2020, Frankfurt am Main, Germany, 06/21/20. https://doi.org/10.1007/978-3-030-59851-8_14

Ensembles of Networks Produced from Neural Architecture Search. / Herron, Emily J.; Young, Steven R.; Potok, Thomas E.
High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers. ed. / Heike Jagode; Hartwig Anzt; Guido Juckeland; Hatem Ltaief. Springer Science and Business Media Deutschland GmbH, 2020. p. 223-234 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 12321 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Ensembles of Networks Produced from Neural Architecture Search

AU - Herron, Emily J.

AU - Young, Steven R.

AU - Potok, Thomas E.

PY - 2020

Y1 - 2020

N2 - Neural architecture search (NAS) is a popular topic at the intersection of deep learning and high performance computing. NAS focuses on optimizing the architecture of neural networks along with their hyperparameters in order to produce networks with superior performance. Much of the focus has been on how to produce a single best network to solve a machine learning problem, but as NAS methods produce many networks that work very well, this affords the opportunity to ensemble these networks to produce an improved result. Additionally, the diversity of network structures produced by NAS drives a natural bias towards diversity of predictions produced by the individual networks. This results in an improved ensemble over simply creating an ensemble that contains duplicates of the best network architecture retrained to have unique weights.

AB - Neural architecture search (NAS) is a popular topic at the intersection of deep learning and high performance computing. NAS focuses on optimizing the architecture of neural networks along with their hyperparameters in order to produce networks with superior performance. Much of the focus has been on how to produce a single best network to solve a machine learning problem, but as NAS methods produce many networks that work very well, this affords the opportunity to ensemble these networks to produce an improved result. Additionally, the diversity of network structures produced by NAS drives a natural bias towards diversity of predictions produced by the individual networks. This results in an improved ensemble over simply creating an ensemble that contains duplicates of the best network architecture retrained to have unique weights.

KW - Ensembles

KW - High performance computing

KW - Neural architecture search

UR - http://www.scopus.com/inward/record.url?scp=85096426440&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-59851-8_14

DO - 10.1007/978-3-030-59851-8_14

M3 - Conference contribution

AN - SCOPUS:85096426440

SN - 9783030598501

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 223

EP - 234

BT - High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers

A2 - Jagode, Heike

A2 - Anzt, Hartwig

A2 - Juckeland, Guido

A2 - Ltaief, Hatem

PB - Springer Science and Business Media Deutschland GmbH

T2 - 35th International Conference on High Performance Computing , ISC High Performance 2020

Y2 - 21 June 2020 through 25 June 2020

ER -

Herron EJ , Young SR , Potok TE. Ensembles of Networks Produced from Neural Architecture Search. In Jagode H, Anzt H, Juckeland G, Ltaief H, editors, High Performance Computing - ISC High Performance 2020 International Workshops, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2020. p. 223-234. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-59851-8_14

Ensembles of Networks Produced from Neural Architecture Search

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this