Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes

Mark Coletti; Dalton Lunga; Anne Berres; Jibonananda Sanyal; Amy Rose

doi:10.1109/MLHPC.2018.8638644

Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes

Mark Coletti, Dalton Lunga, Anne Berres, Jibonananda Sanyal, Amy Rose

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Deep-learners have many hyper-parameters including learning rate, batch size, kernel size - all playing a significant role toward estimating high quality models. Discovering useful hyper-parameter guidelines is an active area of research, though the state of the art generally uses a brute force, uniform grid approach or random search for finding ideal settings. We share the preliminary results of using an alternative approach to deep learner hyper-parameter tuning that uses an evolutionary algorithm to improve the accuracy of a deep-learner models used in satellite imagery building footprint detection. We found that the kernel and batch size hyper-parameters surprisingly differed from sizes arrived at via a brute force uniform grid approach. These differences suggest a novel role for evolutionary algorithms in determining the number of convolution layers, as well as smaller batch sizes in improving deep-learner models.

Original language	English
Title of host publication	Proceedings of MLHPC 2018
Subtitle of host publication	Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	106-113
Number of pages	8
ISBN (Electronic)	9781728101804
DOIs	https://doi.org/10.1109/MLHPC.2018.8638644
State	Published - Jul 2 2018
Event	2018 IEEE/ACM Machine Learning in HPC Environments, MLHPC 2018 - Dallas, United States Duration: Nov 12 2018 → …

Publication series

Name	Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	2018 IEEE/ACM Machine Learning in HPC Environments, MLHPC 2018
Country/Territory	United States
City	Dallas
Period	11/12/18 → …

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). VI. ACKNOWLEDGEMENTS Thanks to Jeanette Weaver of ORNL for supplying the building detection image used for Fig. 1. Thanks to Dalton Lunga, also of ORNL, for supplying the training and validation data as well as the original architecture with attendant source code. We would also like to thank the ASCR Leadership Computing Challenge (ALCC) for the 25 million core hours on Titan that made this research possible. This research was funded by the Federal Emergency Management Association (FEMA), the Bill and Melinda Gates Foundation, and via undisclosed federal funding sources.

Funders	Funder number
U.S. Department of Energy
Bill and Melinda Gates Foundation

Keywords

Convolutional neural networks
Deep learning
Evolutionary algorithms
Hyper-parameters
Optimization
Satellite imagery
Settlement detection

Access to Document

10.1109/MLHPC.2018.8638644

Cite this

Coletti, M., Lunga, D., Berres, A., Sanyal, J., & Rose, A. (2018). Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes. In Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 106-113). Article 8638644 (Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MLHPC.2018.8638644

Coletti, Mark ; Lunga, Dalton ; Berres, Anne et al. / Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes. Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 106-113 (Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{fa9e5b612c094ae0a621f3f051d25e03,

title = "Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes",

abstract = "Deep-learners have many hyper-parameters including learning rate, batch size, kernel size - all playing a significant role toward estimating high quality models. Discovering useful hyper-parameter guidelines is an active area of research, though the state of the art generally uses a brute force, uniform grid approach or random search for finding ideal settings. We share the preliminary results of using an alternative approach to deep learner hyper-parameter tuning that uses an evolutionary algorithm to improve the accuracy of a deep-learner models used in satellite imagery building footprint detection. We found that the kernel and batch size hyper-parameters surprisingly differed from sizes arrived at via a brute force uniform grid approach. These differences suggest a novel role for evolutionary algorithms in determining the number of convolution layers, as well as smaller batch sizes in improving deep-learner models.",

keywords = "Convolutional neural networks, Deep learning, Evolutionary algorithms, Hyper-parameters, Optimization, Satellite imagery, Settlement detection",

author = "Mark Coletti and Dalton Lunga and Anne Berres and Jibonananda Sanyal and Amy Rose",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 IEEE/ACM Machine Learning in HPC Environments, MLHPC 2018 ; Conference date: 12-11-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/MLHPC.2018.8638644",

language = "English",

series = "Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "106--113",

booktitle = "Proceedings of MLHPC 2018",

}

Coletti, M , Lunga, D, Berres, A, Sanyal, J & Rose, A 2018, Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes. in Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis., 8638644, Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 106-113, 2018 IEEE/ACM Machine Learning in HPC Environments, MLHPC 2018, Dallas, United States, 11/12/18. https://doi.org/10.1109/MLHPC.2018.8638644

Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes. / Coletti, Mark ; Lunga, Dalton; Berres, Anne et al.
Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2018. p. 106-113 8638644 (Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes

AU - Coletti, Mark

AU - Lunga, Dalton

AU - Berres, Anne

AU - Sanyal, Jibonananda

AU - Rose, Amy

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Deep-learners have many hyper-parameters including learning rate, batch size, kernel size - all playing a significant role toward estimating high quality models. Discovering useful hyper-parameter guidelines is an active area of research, though the state of the art generally uses a brute force, uniform grid approach or random search for finding ideal settings. We share the preliminary results of using an alternative approach to deep learner hyper-parameter tuning that uses an evolutionary algorithm to improve the accuracy of a deep-learner models used in satellite imagery building footprint detection. We found that the kernel and batch size hyper-parameters surprisingly differed from sizes arrived at via a brute force uniform grid approach. These differences suggest a novel role for evolutionary algorithms in determining the number of convolution layers, as well as smaller batch sizes in improving deep-learner models.

AB - Deep-learners have many hyper-parameters including learning rate, batch size, kernel size - all playing a significant role toward estimating high quality models. Discovering useful hyper-parameter guidelines is an active area of research, though the state of the art generally uses a brute force, uniform grid approach or random search for finding ideal settings. We share the preliminary results of using an alternative approach to deep learner hyper-parameter tuning that uses an evolutionary algorithm to improve the accuracy of a deep-learner models used in satellite imagery building footprint detection. We found that the kernel and batch size hyper-parameters surprisingly differed from sizes arrived at via a brute force uniform grid approach. These differences suggest a novel role for evolutionary algorithms in determining the number of convolution layers, as well as smaller batch sizes in improving deep-learner models.

KW - Convolutional neural networks

KW - Deep learning

KW - Evolutionary algorithms

KW - Hyper-parameters

KW - Optimization

KW - Satellite imagery

KW - Settlement detection

UR - http://www.scopus.com/inward/record.url?scp=85063027712&partnerID=8YFLogxK

U2 - 10.1109/MLHPC.2018.8638644

DO - 10.1109/MLHPC.2018.8638644

M3 - Conference contribution

AN - SCOPUS:85063027712

T3 - Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 106

EP - 113

BT - Proceedings of MLHPC 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 IEEE/ACM Machine Learning in HPC Environments, MLHPC 2018

Y2 - 12 November 2018

ER -

Coletti M , Lunga D, Berres A, Sanyal J, Rose A. Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes. In Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2018. p. 106-113. 8638644. (Proceedings of MLHPC 2018: Machine Learning in HPC Environments, Held in conjunction with SC 2018: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/MLHPC.2018.8638644

Ramifications of Evolving Misbehaving Convolutional Neural Network Kernel and Batch Sizes

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this