Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering

Piotr Luszczek; Wissam M. Sid-Lakhdar; Jack Dongarra

doi:10.1177/10943420231166365

Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering

Piotr Luszczek, Wissam M. Sid-Lakhdar, Jack Dongarra

Computer Science and Mathematics Div

Research output: Contribution to journal › Article › peer-review

4 Scopus citations

Abstract

We combine deep Gaussian processes (DGPs) with multitask and transfer learning for the performance modeling and optimization of HPC applications. Deep Gaussian processes merge the uncertainty quantification advantage of Gaussian processes (GPs) with the predictive power of deep learning. Multitask and transfer learning allow for improved learning efficiency when several similar tasks are to be learned simultaneously and when previous learned models are sought to help in the learning of new tasks, respectively. A comparison with state-of-the-art autotuners shows the advantage of our approach on two application problems. In this article, we combine DGPs with multitask and transfer learning to allow for both an improved tuning of an application parameters on problems of interest but also the prediction of parameters on any potential problem the application might encounter.

Original language	English
Pages (from-to)	229-244
Number of pages	16
Journal	International Journal of High Performance Computing Applications
Volume	37
Issue number	3-4
DOIs	https://doi.org/10.1177/10943420231166365
State	Published - Jul 2023

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The surrogate performance model portion of this work was supported by the U.S. Department of Energy, Office of Science, and ASCR under Award Number DE-SC0021419. The development of some of the software libraries tested in this work was supported by the National Science Foundation Office of Advanced Cyberinfrastructure Directorate for Comp. & Info. Sci. & Eng. under Grant No. 2004541. This research and software used in this work was also supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration, and by the U.S. Department of Energy and Office of Science, under Contracts DE-AC05-00OR22725 and DE-AC52-07NA27344.

Keywords

Efficient Global Optimization
Gaussian process regression
Latin Hypercube Sampling
Linear Coregionalization Model
performance autotuning

Access to Document

10.1177/10943420231166365

Cite this

@article{10494d831d304b8593c1d49c9cbad877,

title = "Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering",

abstract = "We combine deep Gaussian processes (DGPs) with multitask and transfer learning for the performance modeling and optimization of HPC applications. Deep Gaussian processes merge the uncertainty quantification advantage of Gaussian processes (GPs) with the predictive power of deep learning. Multitask and transfer learning allow for improved learning efficiency when several similar tasks are to be learned simultaneously and when previous learned models are sought to help in the learning of new tasks, respectively. A comparison with state-of-the-art autotuners shows the advantage of our approach on two application problems. In this article, we combine DGPs with multitask and transfer learning to allow for both an improved tuning of an application parameters on problems of interest but also the prediction of parameters on any potential problem the application might encounter.",

keywords = "Efficient Global Optimization, Gaussian process regression, Latin Hypercube Sampling, Linear Coregionalization Model, performance autotuning",

author = "Piotr Luszczek and Sid-Lakhdar, {Wissam M.} and Jack Dongarra",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2023.",

year = "2023",

month = jul,

doi = "10.1177/10943420231166365",

language = "English",

volume = "37",

pages = "229--244",

journal = "International Journal of High Performance Computing Applications",

issn = "1094-3420",

publisher = "SAGE Publications",

number = "3-4",

}

TY - JOUR

T1 - Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering

AU - Luszczek, Piotr

AU - Sid-Lakhdar, Wissam M.

AU - Dongarra, Jack

N1 - Publisher Copyright: © The Author(s) 2023.

PY - 2023/7

Y1 - 2023/7

N2 - We combine deep Gaussian processes (DGPs) with multitask and transfer learning for the performance modeling and optimization of HPC applications. Deep Gaussian processes merge the uncertainty quantification advantage of Gaussian processes (GPs) with the predictive power of deep learning. Multitask and transfer learning allow for improved learning efficiency when several similar tasks are to be learned simultaneously and when previous learned models are sought to help in the learning of new tasks, respectively. A comparison with state-of-the-art autotuners shows the advantage of our approach on two application problems. In this article, we combine DGPs with multitask and transfer learning to allow for both an improved tuning of an application parameters on problems of interest but also the prediction of parameters on any potential problem the application might encounter.

AB - We combine deep Gaussian processes (DGPs) with multitask and transfer learning for the performance modeling and optimization of HPC applications. Deep Gaussian processes merge the uncertainty quantification advantage of Gaussian processes (GPs) with the predictive power of deep learning. Multitask and transfer learning allow for improved learning efficiency when several similar tasks are to be learned simultaneously and when previous learned models are sought to help in the learning of new tasks, respectively. A comparison with state-of-the-art autotuners shows the advantage of our approach on two application problems. In this article, we combine DGPs with multitask and transfer learning to allow for both an improved tuning of an application parameters on problems of interest but also the prediction of parameters on any potential problem the application might encounter.

KW - Efficient Global Optimization

KW - Gaussian process regression

KW - Latin Hypercube Sampling

KW - Linear Coregionalization Model

KW - performance autotuning

UR - http://www.scopus.com/inward/record.url?scp=85152405843&partnerID=8YFLogxK

U2 - 10.1177/10943420231166365

DO - 10.1177/10943420231166365

M3 - Article

AN - SCOPUS:85152405843

SN - 1094-3420

VL - 37

SP - 229

EP - 244

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

IS - 3-4

ER -

Combining multitask and transfer learning with deep Gaussian processes for autotuning-based performance engineering

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this