Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach

Maryam Parsa; Catherine Schuman; Nitin Rathi; Amir Ziabari; Derek Rose; J. Parker Mitchell; J. Travis Johnston; Bill Kay; Steven Young; Kaushik Roy

doi:10.1145/3477145.3477160

Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach

Maryam Parsa, Catherine Schuman, Nitin Rathi, Amir Ziabari, Derek Rose, J. Parker Mitchell, J. Travis Johnston, Bill Kay, Steven Young, Kaushik Roy

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Neuromorphic systems allow for extremely efficient hardware implementations for neural networks (NNs). In recent years, several algorithms have been presented to train spiking NNs (SNNs) for neuromorphic hardware. However, SNNs often provide lower accuracy than their artificial NNs (ANNs) counterparts or require computationally expensive and slow training/inference methods. To close this gap, designers typically rely on reconfiguring SNNs through adjustments in the neuron/synapse model or training algorithm itself. Nevertheless, these steps incur significant design time, while still lacking the desired improvement in terms of training/inference times (latency). Designing SNNs that can mimic the accuracy of ANNs with reasonable training times is an exigent challenge in neuromorphic computing. In this work, we present an alternative approach that looks at such designs as an optimization problem rather than algorithm or architecture redesign. We develop a versatile multiobjective hyperparameter optimization (HPO) for automatically tuning HPs of two state-of-The-Art SNN training algorithms, SLAYER and HYBRID. We emphasize that, to the best of our knowledge, this is the first work trying to improve SNNs' computational efficiency, accuracy, and training time using an efficient HPO. We demonstrate significant performance improvements for SNNs on several datasets without the need to redesign or invent new training algorithms/architectures. Our approach results in more accurate networks with lower latency and, in turn, higher energy efficiency than previous implementations. In particular, we demonstrate improvement in accuracy and more than 5 × reduction in the training/inference time for the SLAYER algorithm on the DVS Gesture dataset. In the case of HYBRID, we demonstrate 30% reduction in timesteps while surpassing the accuracy of the state-of-The-Art networks on CIFAR10. Further, our analysis suggests that even a seemingly minor change in HPs could change the accuracy by 5-6 ×.

Original language	English
Title of host publication	ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021
Publisher	Association for Computing Machinery
ISBN (Electronic)	9781450386913
DOIs	https://doi.org/10.1145/3477145.3477160
State	Published - Jul 27 2021
Event	2021 International Conference on Neuromorphic Systems, ICONS 2021 - Virtual, Onlie, United States Duration: Jul 27 2021 → Jul 29 2021

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2021 International Conference on Neuromorphic Systems, ICONS 2021
Country/Territory	United States
City	Virtual, Onlie
Period	07/27/21 → 07/29/21

Funding

Notice: This manuscript has been authored in part by Center for Brain Inspired Computing Enabling Autonomous Intelligence (C-BRIC), one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, the National Science Foundation, Intel Corporation and Vannevar Bush Faculty Fellowship, and UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, under contract number DE-AC05-00OR22725, and in part by the Laboratory Directed Research and Development Program of Oak Ridge National Laboratory, managed by UT-Battelle, LLC. The research is also funded in part by Center for Brain Inspired Computing Enabling Autonomous Intelligence (C-BRIC), one of six centers in JUMP, a Semiconductor Research Corporation (SRC) program sponsored by DARPA, the National Science Foundation, Intel Corporation and Vannevar Bush Faculty Fellowship.

Keywords

Bayesian optimization
Neuromorphic computing
hyperparameter optimization
multiobjective optimization
spiking neural networks

Access to Document

10.1145/3477145.3477160

Cite this

Parsa, M., Schuman, C., Rathi, N., Ziabari, A., Rose, D., Mitchell, J. P., Johnston, J. T., Kay, B., Young, S., & Roy, K. (2021). Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach. In ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021 Article 3477160 (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3477145.3477160

@inproceedings{f6ddd80e064e465f97d4a83c8c9c2489,

title = "Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach",

abstract = "Neuromorphic systems allow for extremely efficient hardware implementations for neural networks (NNs). In recent years, several algorithms have been presented to train spiking NNs (SNNs) for neuromorphic hardware. However, SNNs often provide lower accuracy than their artificial NNs (ANNs) counterparts or require computationally expensive and slow training/inference methods. To close this gap, designers typically rely on reconfiguring SNNs through adjustments in the neuron/synapse model or training algorithm itself. Nevertheless, these steps incur significant design time, while still lacking the desired improvement in terms of training/inference times (latency). Designing SNNs that can mimic the accuracy of ANNs with reasonable training times is an exigent challenge in neuromorphic computing. In this work, we present an alternative approach that looks at such designs as an optimization problem rather than algorithm or architecture redesign. We develop a versatile multiobjective hyperparameter optimization (HPO) for automatically tuning HPs of two state-of-The-Art SNN training algorithms, SLAYER and HYBRID. We emphasize that, to the best of our knowledge, this is the first work trying to improve SNNs' computational efficiency, accuracy, and training time using an efficient HPO. We demonstrate significant performance improvements for SNNs on several datasets without the need to redesign or invent new training algorithms/architectures. Our approach results in more accurate networks with lower latency and, in turn, higher energy efficiency than previous implementations. In particular, we demonstrate improvement in accuracy and more than 5 × reduction in the training/inference time for the SLAYER algorithm on the DVS Gesture dataset. In the case of HYBRID, we demonstrate 30% reduction in timesteps while surpassing the accuracy of the state-of-The-Art networks on CIFAR10. Further, our analysis suggests that even a seemingly minor change in HPs could change the accuracy by 5-6 ×.",

keywords = "Bayesian optimization, Neuromorphic computing, hyperparameter optimization, multiobjective optimization, spiking neural networks",

author = "Maryam Parsa and Catherine Schuman and Nitin Rathi and Amir Ziabari and Derek Rose and Mitchell, {J. Parker} and Johnston, {J. Travis} and Bill Kay and Steven Young and Kaushik Roy",

note = "Publisher Copyright: {\textcopyright} 2021 ACM.; 2021 International Conference on Neuromorphic Systems, ICONS 2021 ; Conference date: 27-07-2021 Through 29-07-2021",

year = "2021",

month = jul,

day = "27",

doi = "10.1145/3477145.3477160",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

booktitle = "ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021",

}

Parsa, M, Schuman, C, Rathi, N, Ziabari, A , Rose, D, Mitchell, JP, Johnston, JT, Kay, B, Young, S & Roy, K 2021, Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach. in ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021., 3477160, ACM International Conference Proceeding Series, Association for Computing Machinery, 2021 International Conference on Neuromorphic Systems, ICONS 2021, Virtual, Onlie, United States, 07/27/21. https://doi.org/10.1145/3477145.3477160

Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach. / Parsa, Maryam; Schuman, Catherine; Rathi, Nitin et al.
ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021. Association for Computing Machinery, 2021. 3477160 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach

AU - Parsa, Maryam

AU - Schuman, Catherine

AU - Rathi, Nitin

AU - Ziabari, Amir

AU - Rose, Derek

AU - Mitchell, J. Parker

AU - Johnston, J. Travis

AU - Kay, Bill

AU - Young, Steven

AU - Roy, Kaushik

PY - 2021/7/27

Y1 - 2021/7/27

N2 - Neuromorphic systems allow for extremely efficient hardware implementations for neural networks (NNs). In recent years, several algorithms have been presented to train spiking NNs (SNNs) for neuromorphic hardware. However, SNNs often provide lower accuracy than their artificial NNs (ANNs) counterparts or require computationally expensive and slow training/inference methods. To close this gap, designers typically rely on reconfiguring SNNs through adjustments in the neuron/synapse model or training algorithm itself. Nevertheless, these steps incur significant design time, while still lacking the desired improvement in terms of training/inference times (latency). Designing SNNs that can mimic the accuracy of ANNs with reasonable training times is an exigent challenge in neuromorphic computing. In this work, we present an alternative approach that looks at such designs as an optimization problem rather than algorithm or architecture redesign. We develop a versatile multiobjective hyperparameter optimization (HPO) for automatically tuning HPs of two state-of-The-Art SNN training algorithms, SLAYER and HYBRID. We emphasize that, to the best of our knowledge, this is the first work trying to improve SNNs' computational efficiency, accuracy, and training time using an efficient HPO. We demonstrate significant performance improvements for SNNs on several datasets without the need to redesign or invent new training algorithms/architectures. Our approach results in more accurate networks with lower latency and, in turn, higher energy efficiency than previous implementations. In particular, we demonstrate improvement in accuracy and more than 5 × reduction in the training/inference time for the SLAYER algorithm on the DVS Gesture dataset. In the case of HYBRID, we demonstrate 30% reduction in timesteps while surpassing the accuracy of the state-of-The-Art networks on CIFAR10. Further, our analysis suggests that even a seemingly minor change in HPs could change the accuracy by 5-6 ×.

AB - Neuromorphic systems allow for extremely efficient hardware implementations for neural networks (NNs). In recent years, several algorithms have been presented to train spiking NNs (SNNs) for neuromorphic hardware. However, SNNs often provide lower accuracy than their artificial NNs (ANNs) counterparts or require computationally expensive and slow training/inference methods. To close this gap, designers typically rely on reconfiguring SNNs through adjustments in the neuron/synapse model or training algorithm itself. Nevertheless, these steps incur significant design time, while still lacking the desired improvement in terms of training/inference times (latency). Designing SNNs that can mimic the accuracy of ANNs with reasonable training times is an exigent challenge in neuromorphic computing. In this work, we present an alternative approach that looks at such designs as an optimization problem rather than algorithm or architecture redesign. We develop a versatile multiobjective hyperparameter optimization (HPO) for automatically tuning HPs of two state-of-The-Art SNN training algorithms, SLAYER and HYBRID. We emphasize that, to the best of our knowledge, this is the first work trying to improve SNNs' computational efficiency, accuracy, and training time using an efficient HPO. We demonstrate significant performance improvements for SNNs on several datasets without the need to redesign or invent new training algorithms/architectures. Our approach results in more accurate networks with lower latency and, in turn, higher energy efficiency than previous implementations. In particular, we demonstrate improvement in accuracy and more than 5 × reduction in the training/inference time for the SLAYER algorithm on the DVS Gesture dataset. In the case of HYBRID, we demonstrate 30% reduction in timesteps while surpassing the accuracy of the state-of-The-Art networks on CIFAR10. Further, our analysis suggests that even a seemingly minor change in HPs could change the accuracy by 5-6 ×.

KW - Bayesian optimization

KW - Neuromorphic computing

KW - hyperparameter optimization

KW - multiobjective optimization

KW - spiking neural networks

UR - http://www.scopus.com/inward/record.url?scp=85117950189&partnerID=8YFLogxK

U2 - 10.1145/3477145.3477160

DO - 10.1145/3477145.3477160

M3 - Conference contribution

AN - SCOPUS:85117950189

T3 - ACM International Conference Proceeding Series

BT - ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021

PB - Association for Computing Machinery

T2 - 2021 International Conference on Neuromorphic Systems, ICONS 2021

Y2 - 27 July 2021 through 29 July 2021

ER -

Parsa M, Schuman C, Rathi N, Ziabari A , Rose D, Mitchell JP et al. Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach. In ICONS 2021 - Proceedings of International Conference on Neuromorphic Systems 2021. Association for Computing Machinery. 2021. 3477160. (ACM International Conference Proceeding Series). doi: 10.1145/3477145.3477160

Accurate and Accelerated Neuromorphic Network Design Leveraging A Bayesian Hyperparameter Pareto Optimization Approach

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this