Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation

Pedro Valero-Lara; Alexis Huante; Mustafa Al Lail; William F. Godoy; Keita Teranishi; Prasanna Balaprakash; Jeffrey S. Vetter

doi:10.1007/978-3-032-02436-7_2

Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.

Original language	English
Title of host publication	Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers
Editors	Henry Dietz
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	20-32
Number of pages	13
ISBN (Print)	9783032024350
DOIs	https://doi.org/10.1007/978-3-032-02436-7_2
State	Published - 2026
Event	36th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2023 - Lexington, United States Duration: Oct 11 2023 → Oct 13 2023

Publication series

Name	Lecture Notes in Computer Science
Volume	14480 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	36th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2023
Country/Territory	United States
City	Lexington
Period	10/11/23 → 10/13/23

Funding

This work is funded by Bluestone, an X-Stack project in the DOE Advanced Scientific Computing Office with program manager Hal Finkel.

Keywords

GPT
HPC
LLM
Llama-2

Access to Document

10.1007/978-3-032-02436-7_2

Cite this

Valero-Lara, P., Huante, A., Al Lail, M., Godoy, W. F., Teranishi, K., Balaprakash, P., & Vetter, J. S. (2026). Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation. In H. Dietz (Ed.), Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers (pp. 20-32). (Lecture Notes in Computer Science; Vol. 14480 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-032-02436-7_2

@inproceedings{11ec14efe4504432ab92185de70dd85c,

title = "Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation",

abstract = "We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.",

keywords = "GPT, HPC, LLM, Llama-2",

author = "Pedro Valero-Lara and Alexis Huante and \{Al Lail\}, Mustafa and Godoy, \{William F.\} and Keita Teranishi and Prasanna Balaprakash and Vetter, \{Jeffrey S.\}",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.; 36th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2023 ; Conference date: 11-10-2023 Through 13-10-2023",

year = "2026",

doi = "10.1007/978-3-032-02436-7\_2",

language = "English",

isbn = "9783032024350",

series = "Lecture Notes in Computer Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "20--32",

editor = "Henry Dietz",

booktitle = "Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers",

}

Valero-Lara, P, Huante, A, Al Lail, M, Godoy, WF , Teranishi, K , Balaprakash, P & Vetter, JS 2026, Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation. in H Dietz (ed.), Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers. Lecture Notes in Computer Science, vol. 14480 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 20-32, 36th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2023, Lexington, United States, 10/11/23. https://doi.org/10.1007/978-3-032-02436-7_2

Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation. / Valero-Lara, Pedro; Huante, Alexis; Al Lail, Mustafa et al.
Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers. ed. / Henry Dietz. Springer Science and Business Media Deutschland GmbH, 2026. p. 20-32 (Lecture Notes in Computer Science; Vol. 14480 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation

AU - Valero-Lara, Pedro

AU - Huante, Alexis

AU - Al Lail, Mustafa

AU - Godoy, William F.

AU - Teranishi, Keita

AU - Balaprakash, Prasanna

AU - Vetter, Jeffrey S.

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2026.

PY - 2026

Y1 - 2026

N2 - We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.

AB - We evaluate the use of the open-source Llama-2 model for generating well-known, high-performance computing kernels (e.g., AXPY, GEMV, GEMM) on different parallel programming models and languages (e.g., C++: OpenMP, OpenMP Offload, OpenACC, CUDA, HIP; Fortran: OpenMP, OpenMP Offload, OpenACC; Python: numpy, Numba, pyCUDA, cuPy; and Julia: Threads, CUDA.jl, AMDGPU.jl). We built upon our previous work that is based on the OpenAI Codex, which is a descendant of GPT-3, to generate similar kernels with simple prompts via GitHub Copilot. Our goal is to compare the accuracy of Llama-2 and our original GPT-3 baseline by using a similar metric. Llama-2 has a simplified model that shows competitive or even superior accuracy. We also report on the differences between these foundational large language models as generative AI continues to redefine human-computer interactions. Overall, Copilot generates codes that are more reliable but less optimized, whereas codes generated by Llama-2 are less reliable but more optimized when correct.

KW - GPT

KW - HPC

KW - LLM

KW - Llama-2

UR - https://www.scopus.com/pages/publications/105021402229

U2 - 10.1007/978-3-032-02436-7_2

DO - 10.1007/978-3-032-02436-7_2

M3 - Conference contribution

AN - SCOPUS:105021402229

SN - 9783032024350

T3 - Lecture Notes in Computer Science

SP - 20

EP - 32

BT - Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers

A2 - Dietz, Henry

PB - Springer Science and Business Media Deutschland GmbH

T2 - 36th International Workshop on Languages and Compilers for Parallel Computing, LCPC 2023

Y2 - 11 October 2023 through 13 October 2023

ER -

Valero-Lara P, Huante A, Al Lail M, Godoy WF , Teranishi K , Balaprakash P et al. Comparing Llama-2 and GPT-3 LLMs for HPC Kernels Generation. In Dietz H, editor, Languages and Compilers for Parallel Computing - 36th International Workshop, LCPC 2023, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2026. p. 20-32. (Lecture Notes in Computer Science). doi: 10.1007/978-3-032-02436-7_2