Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks

Yaohung M. Tsai; Piotr Luszczek; Jakub Kurzak; Jack Dongarra

doi:10.1109/MLHPC.2016.5

Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks

Yaohung M. Tsai, Piotr Luszczek, Jakub Kurzak, Jack Dongarra

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

We present a portable and highly-optimized Deep Neural Network (DNN) algorithm and its implementation techniques. Our approach is a novel combination of existing HPC techniques that methodically applies autotuning as well as data layout and low-level optimizations that achieve performance matching and/or exceeding what is possible with either reverse engineering and manual assembly coding or proprietary vendor libraries. The former was done inside the maxDNN implementation and the latter is represented by cuDNN. Our work may be directly applied to the most time consuming part of DNN workflow, namely the training process which often needs a restart when it stagnates due to, for example, diminishing gradients and getting stuck in local minima. With the result of performance tests on a consumer-grade GPU with the latest High Bandwidth Memory (HBM) stack, our methodology can match a server grade hardware at a fraction of the price. Another tuning sweep on a new GPU architecture from a different vendor also attests to the portability of our approach and the quality of our implementation.

Original language	English
Title of host publication	Proceedings of MLHPC 2016
Subtitle of host publication	Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	9-18
Number of pages	10
ISBN (Electronic)	9781509038824
DOIs	https://doi.org/10.1109/MLHPC.2016.5
State	Published - Jan 27 2017
Externally published	Yes
Event	2016 Machine Learning in HPC Environments, MLHPC 2016 - Salt Lake City, United States Duration: Nov 14 2016 → …

Publication series

Name	Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	2016 Machine Learning in HPC Environments, MLHPC 2016
Country/Territory	United States
City	Salt Lake City
Period	11/14/16 → …

Funding

This work was funded by NSF through Award Number 1439052. We would like to thank the BEAST team for help with autotuning sweeps.

Access to Document

10.1109/MLHPC.2016.5

Cite this

Tsai, Y. M., Luszczek, P., Kurzak, J., & Dongarra, J. (2017). Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks. In Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 9-18). Article 7835790 (Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MLHPC.2016.5

Tsai, Yaohung M. ; Luszczek, Piotr ; Kurzak, Jakub et al. / Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks. Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 9-18 (Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{dc5cf97b503d424d91887356be5cbf1b,

title = "Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks",

abstract = "We present a portable and highly-optimized Deep Neural Network (DNN) algorithm and its implementation techniques. Our approach is a novel combination of existing HPC techniques that methodically applies autotuning as well as data layout and low-level optimizations that achieve performance matching and/or exceeding what is possible with either reverse engineering and manual assembly coding or proprietary vendor libraries. The former was done inside the maxDNN implementation and the latter is represented by cuDNN. Our work may be directly applied to the most time consuming part of DNN workflow, namely the training process which often needs a restart when it stagnates due to, for example, diminishing gradients and getting stuck in local minima. With the result of performance tests on a consumer-grade GPU with the latest High Bandwidth Memory (HBM) stack, our methodology can match a server grade hardware at a fraction of the price. Another tuning sweep on a new GPU architecture from a different vendor also attests to the portability of our approach and the quality of our implementation.",

author = "Tsai, {Yaohung M.} and Piotr Luszczek and Jakub Kurzak and Jack Dongarra",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 Machine Learning in HPC Environments, MLHPC 2016 ; Conference date: 14-11-2016",

year = "2017",

month = jan,

day = "27",

doi = "10.1109/MLHPC.2016.5",

language = "English",

series = "Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "9--18",

booktitle = "Proceedings of MLHPC 2016",

}

Tsai, YM, Luszczek, P, Kurzak, J & Dongarra, J 2017, Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks. in Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis., 7835790, Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 9-18, 2016 Machine Learning in HPC Environments, MLHPC 2016, Salt Lake City, United States, 11/14/16. https://doi.org/10.1109/MLHPC.2016.5

Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks. / Tsai, Yaohung M.; Luszczek, Piotr; Kurzak, Jakub et al.
Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2017. p. 9-18 7835790 (Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks

AU - Tsai, Yaohung M.

AU - Luszczek, Piotr

AU - Kurzak, Jakub

AU - Dongarra, Jack

PY - 2017/1/27

Y1 - 2017/1/27

N2 - We present a portable and highly-optimized Deep Neural Network (DNN) algorithm and its implementation techniques. Our approach is a novel combination of existing HPC techniques that methodically applies autotuning as well as data layout and low-level optimizations that achieve performance matching and/or exceeding what is possible with either reverse engineering and manual assembly coding or proprietary vendor libraries. The former was done inside the maxDNN implementation and the latter is represented by cuDNN. Our work may be directly applied to the most time consuming part of DNN workflow, namely the training process which often needs a restart when it stagnates due to, for example, diminishing gradients and getting stuck in local minima. With the result of performance tests on a consumer-grade GPU with the latest High Bandwidth Memory (HBM) stack, our methodology can match a server grade hardware at a fraction of the price. Another tuning sweep on a new GPU architecture from a different vendor also attests to the portability of our approach and the quality of our implementation.

AB - We present a portable and highly-optimized Deep Neural Network (DNN) algorithm and its implementation techniques. Our approach is a novel combination of existing HPC techniques that methodically applies autotuning as well as data layout and low-level optimizations that achieve performance matching and/or exceeding what is possible with either reverse engineering and manual assembly coding or proprietary vendor libraries. The former was done inside the maxDNN implementation and the latter is represented by cuDNN. Our work may be directly applied to the most time consuming part of DNN workflow, namely the training process which often needs a restart when it stagnates due to, for example, diminishing gradients and getting stuck in local minima. With the result of performance tests on a consumer-grade GPU with the latest High Bandwidth Memory (HBM) stack, our methodology can match a server grade hardware at a fraction of the price. Another tuning sweep on a new GPU architecture from a different vendor also attests to the portability of our approach and the quality of our implementation.

UR - http://www.scopus.com/inward/record.url?scp=85015243309&partnerID=8YFLogxK

U2 - 10.1109/MLHPC.2016.5

DO - 10.1109/MLHPC.2016.5

M3 - Conference contribution

AN - SCOPUS:85015243309

T3 - Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 9

EP - 18

BT - Proceedings of MLHPC 2016

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2016 Machine Learning in HPC Environments, MLHPC 2016

Y2 - 14 November 2016

ER -

Tsai YM, Luszczek P, Kurzak J, Dongarra J. Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks. In Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2017. p. 9-18. 7835790. (Proceedings of MLHPC 2016: Machine Learning in HPC Environments - Held in conjunction with SC 2016: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/MLHPC.2016.5

Performance-portable autotuning of OpenCL kernels for convolutional layers of deep neural networks

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this