Training deep neural networks with constrained learning parameters

Prasanna Date, Christopher D. Carothers, John E. Mitchell, James A. Hendler, Malik Magdon-Ismail

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

1 Scopus citations

Abstract

Today's deep learning models are primarily trained on CPUs and GPUs. Although these models tend to have low error, they consume high power and utilize large amount of memory owing to double precision floating point learning parameters. Beyond the Moore's law, a significant portion of deep learning tasks would run on edge computing systems, which will form an indispensable part of the entire computation fabric. Subsequently, training deep learning models for such systems will have to be tailored and adopted to generate models that have the following desirable characteristics: low error, low memory, and low power. We believe that deep neural networks (DNNs), where learning parameters are constrained to have a set of finite discrete values, running on neuromorphic computing systems would be instrumental for intelligent edge computing systems having these desirable characteristics. To this extent, we propose the Combinatorial Neural Network Training Algorithm (CoNNTrA), that leverages a coordinate gradient descent-based approach for training deep learning models with finite discrete learning parameters. Next, we elaborate on the theoretical underpinnings and evaluate the computational complexity of CoNNTrA. As a proof of concept, we use CoNNTrA to train deep learning models with ternary learning parameters on the MNIST, Iris and ImageNet data sets and compare their performance to the same models trained using Backpropagation. We use following performance metrics for the comparison: (i) Training error; (ii) Validation error; (iii) Memory usage; and (iv) Training time. Our results indicate that CoNNTrA models use 32× less memory and have errors at par with the Backpropagation models.

Original languageEnglish
Title of host publicationProceedings - 2020 International Conference on Rebooting Computing, ICRC 2020
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages107-115
Number of pages9
ISBN (Electronic)9780738143378
DOIs
StatePublished - Dec 2020
Externally publishedYes
Event2020 International Conference on Rebooting Computing, ICRC 2020 - Virtual, Atlanta, United States
Duration: Dec 1 2020Dec 3 2020

Publication series

NameProceedings - 2020 International Conference on Rebooting Computing, ICRC 2020

Conference

Conference2020 International Conference on Rebooting Computing, ICRC 2020
Country/TerritoryUnited States
CityVirtual, Atlanta
Period12/1/2012/3/20

Keywords

  • Artificial intelligence
  • Deep learning
  • Deep neural networks
  • Machine learning
  • Training algorithm

Fingerprint

Dive into the research topics of 'Training deep neural networks with constrained learning parameters'. Together they form a unique fingerprint.

Cite this