Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Maryam Parsa; Catherine D. Schuman; Prasanna Date; Derek C. Rose; Bill Kay; J. Parker Mitchell; Steven R. Young; Ryan Dellana; William Severa; Thomas E. Potok; Kaushik Roy

doi:10.1109/IJCNN48605.2020.9206872

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Maryam Parsa, Catherine D. Schuman, Prasanna Date, Derek C. Rose, Bill Kay, J. Parker Mitchell, Steven R. Young, Ryan Dellana, William Severa, Thomas E. Potok, Kaushik Roy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.

Original language	English
Title of host publication	2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9781728169262
DOIs	https://doi.org/10.1109/IJCNN48605.2020.9206872
State	Published - Jul 2020
Event	2020 International Joint Conference on Neural Networks, IJCNN 2020 - Virtual, Glasgow, United Kingdom Duration: Jul 19 2020 → Jul 24 2020

Publication series

Name	Proceedings of the International Joint Conference on Neural Networks

Conference

Conference	2020 International Joint Conference on Neural Networks, IJCNN 2020
Country/Territory	United Kingdom
City	Virtual, Glasgow
Period	07/19/20 → 07/24/20

Funding

This material is also based in part upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC05-00OR22725, and in part by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC. The research was funded in part by Center for Brain-Inspired Computing Enabling Autonomous Intelligence (C-BRIC), one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, the National Science Foundation, Intel Corporation and Vannevar Bush Faculty Fellowship. Sandia National Laboratories is a multimission laboratory managed and operated by National Technology & Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Hon-eywell International Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525. Notice: This manuscript has been authored in part by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

Bayesian optimization
hyperparameter optimization
neural networks
neuromorphic

Access to Document

10.1109/IJCNN48605.2020.9206872

Cite this

Parsa, M., Schuman, C. D., Date, P., Rose, D. C., Kay, B., Mitchell, J. P., Young, S. R., Dellana, R., Severa, W., Potok, T. E., & Roy, K. (2020). Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment. In 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings Article 9206872 (Proceedings of the International Joint Conference on Neural Networks). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IJCNN48605.2020.9206872

Parsa, Maryam ; Schuman, Catherine D. ; Date, Prasanna et al. / Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment. 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. (Proceedings of the International Joint Conference on Neural Networks).

@inproceedings{135c8520acd444f4b75a4c5c579c9e65,

title = "Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment",

abstract = "Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.",

keywords = "Bayesian optimization, hyperparameter optimization, neural networks, neuromorphic",

author = "Maryam Parsa and Schuman, {Catherine D.} and Prasanna Date and Rose, {Derek C.} and Bill Kay and Mitchell, {J. Parker} and Young, {Steven R.} and Ryan Dellana and William Severa and Potok, {Thomas E.} and Kaushik Roy",

note = "Publisher Copyright: {\textcopyright} 2020 IEEE.; 2020 International Joint Conference on Neural Networks, IJCNN 2020 ; Conference date: 19-07-2020 Through 24-07-2020",

year = "2020",

month = jul,

doi = "10.1109/IJCNN48605.2020.9206872",

language = "English",

series = "Proceedings of the International Joint Conference on Neural Networks",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings",

}

Parsa, M, Schuman, CD, Date, P , Rose, DC, Kay, B, Mitchell, JP, Young, SR, Dellana, R, Severa, W, Potok, TE & Roy, K 2020, Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment. in 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings., 9206872, Proceedings of the International Joint Conference on Neural Networks, Institute of Electrical and Electronics Engineers Inc., 2020 International Joint Conference on Neural Networks, IJCNN 2020, Virtual, Glasgow, United Kingdom, 07/19/20. https://doi.org/10.1109/IJCNN48605.2020.9206872

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment. / Parsa, Maryam; Schuman, Catherine D.; Date, Prasanna et al.
2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2020. 9206872 (Proceedings of the International Joint Conference on Neural Networks).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

AU - Parsa, Maryam

AU - Schuman, Catherine D.

AU - Date, Prasanna

AU - Rose, Derek C.

AU - Kay, Bill

AU - Mitchell, J. Parker

AU - Young, Steven R.

AU - Dellana, Ryan

AU - Severa, William

AU - Potok, Thomas E.

AU - Roy, Kaushik

PY - 2020/7

Y1 - 2020/7

N2 - Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.

AB - Training neural networks for neuromorphic deployment is non-trivial. There have been a variety of approaches proposed to adapt back-propagation or back-propagation-like algorithms appropriate for training. Considering that these networks often have very different performance characteristics than traditional neural networks, it is often unclear how to set either the network topology or the hyperparameters to achieve optimal performance. In this work, we introduce a Bayesian approach for optimizing the hyperparameters of an algorithm for training binary communication networks that can be deployed to neuromorphic hardware. We show that by optimizing the hyperparameters on this algorithm for each dataset, we can achieve improvements in accuracy over the previous state-of-the-art for this algorithm on each dataset (by up to 15 percent). This jump in performance continues to emphasize the potential when converting traditional neural networks to binary communication applicable to neuromorphic hardware.

KW - Bayesian optimization

KW - hyperparameter optimization

KW - neural networks

KW - neuromorphic

UR - http://www.scopus.com/inward/record.url?scp=85093866263&partnerID=8YFLogxK

U2 - 10.1109/IJCNN48605.2020.9206872

DO - 10.1109/IJCNN48605.2020.9206872

M3 - Conference contribution

AN - SCOPUS:85093866263

T3 - Proceedings of the International Joint Conference on Neural Networks

BT - 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2020 International Joint Conference on Neural Networks, IJCNN 2020

Y2 - 19 July 2020 through 24 July 2020

ER -

Parsa M, Schuman CD, Date P , Rose DC, Kay B, Mitchell JP et al. Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment. In 2020 International Joint Conference on Neural Networks, IJCNN 2020 - Proceedings. Institute of Electrical and Electronics Engineers Inc. 2020. 9206872. (Proceedings of the International Joint Conference on Neural Networks). doi: 10.1109/IJCNN48605.2020.9206872

Hyperparameter Optimization in Binary Communication Networks for Neuromorphic Deployment

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this