S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document

Kareem Shaik; Dali Wang; Weijian Zheng; Qinglei Cao; Heng Fan; Peter Schwartz; Yunhe Feng

doi:10.1007/978-3-031-63759-9_27

S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document

Kareem Shaik, Dali Wang, Weijian Zheng, Qinglei Cao, Heng Fan, Peter Schwartz, Yunhe Feng

Earth Systems Modeling

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The understanding of large-scale scientific software is a significant challenge due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large language models (LLMs), provides novel pathways for understanding such complex scientific codes. This paper presents S3LLM, an LLM-based framework designed to enable the examination of source code, code metadata, and summarized information in conjunction with textual technical reports in an interactive, conversational manner through a user-friendly interface. S3LLM leverages open-source LLaMA-2 models to enhance code analysis through the automatic transformation of natural language queries into domain-specific language (DSL) queries. In addition, S3LLM is equipped to handle diverse metadata types, including DOT, SQL, and customized formats. Furthermore, S3LLM incorporates retrieval-augmented generation (RAG) and LangChain technologies to directly query extensive documents. S3LLM demonstrates the potential of using locally deployed open-source LLMs for the rapid understanding of large-scale scientific computing software, eliminating the need for extensive coding expertise and thereby making the process more efficient and effective. S3LLM is available at https://github.com/ResponsibleAILab/s3llm.

Original language	English
Title of host publication	Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings
Editors	Leonardo Franco, Clélia de Mulatier, Maciej Paszynski, Valeria V. Krzhizhanovskaya, Jack J. Dongarra, Peter M. A. Sloot
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	222-230
Number of pages	9
ISBN (Print)	9783031637582
DOIs	https://doi.org/10.1007/978-3-031-63759-9_27
State	Published - 2024
Event	24th International Conference on Computational Science, ICCS 2024 - Malaga, Spain Duration: Jul 2 2024 → Jul 4 2024

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	14834 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	24th International Conference on Computational Science, ICCS 2024
Country/Territory	Spain
City	Malaga
Period	07/2/24 → 07/4/24

Funding

This manuscript has been authored by UT-Battelle, LLC under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy. gov/downloads/doe-public-access-plan).

Keywords

ChatGPT
E3SM Land Model
Large-Scale Scientific Software
LLaMA
LLM
Research Software Analysis
Retrieval Augmented Generation (RAG)

Access to Document

10.1007/978-3-031-63759-9_27

Cite this

Shaik, K., Wang, D., Zheng, W., Cao, Q., Fan, H., Schwartz, P., & Feng, Y. (2024). S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. In L. Franco, C. de Mulatier, M. Paszynski, V. V. Krzhizhanovskaya, J. J. Dongarra, & P. M. A. Sloot (Eds.), Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings (pp. 222-230). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14834 LNCS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-63759-9_27

Shaik, Kareem ; Wang, Dali ; Zheng, Weijian et al. / S3LLM : Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings. editor / Leonardo Franco ; Clélia de Mulatier ; Maciej Paszynski ; Valeria V. Krzhizhanovskaya ; Jack J. Dongarra ; Peter M. A. Sloot. Springer Science and Business Media Deutschland GmbH, 2024. pp. 222-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{9df6ee7f3194445b948cfe73c3193e2d,

title = "S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document",

abstract = "The understanding of large-scale scientific software is a significant challenge due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large language models (LLMs), provides novel pathways for understanding such complex scientific codes. This paper presents S3LLM, an LLM-based framework designed to enable the examination of source code, code metadata, and summarized information in conjunction with textual technical reports in an interactive, conversational manner through a user-friendly interface. S3LLM leverages open-source LLaMA-2 models to enhance code analysis through the automatic transformation of natural language queries into domain-specific language (DSL) queries. In addition, S3LLM is equipped to handle diverse metadata types, including DOT, SQL, and customized formats. Furthermore, S3LLM incorporates retrieval-augmented generation (RAG) and LangChain technologies to directly query extensive documents. S3LLM demonstrates the potential of using locally deployed open-source LLMs for the rapid understanding of large-scale scientific computing software, eliminating the need for extensive coding expertise and thereby making the process more efficient and effective. S3LLM is available at https://github.com/ResponsibleAILab/s3llm.",

keywords = "ChatGPT, E3SM Land Model, Large-Scale Scientific Software, LLaMA, LLM, Research Software Analysis, Retrieval Augmented Generation (RAG)",

author = "Kareem Shaik and Dali Wang and Weijian Zheng and Qinglei Cao and Heng Fan and Peter Schwartz and Yunhe Feng",

note = "Publisher Copyright: {\textcopyright} The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.; 24th International Conference on Computational Science, ICCS 2024 ; Conference date: 02-07-2024 Through 04-07-2024",

year = "2024",

doi = "10.1007/978-3-031-63759-9_27",

language = "English",

isbn = "9783031637582",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "222--230",

editor = "Leonardo Franco and {de Mulatier}, Cl{\'e}lia and Maciej Paszynski and Krzhizhanovskaya, {Valeria V.} and Dongarra, {Jack J.} and Sloot, {Peter M. A.}",

booktitle = "Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings",

}

Shaik, K, Wang, D, Zheng, W, Cao, Q, Fan, H, Schwartz, P & Feng, Y 2024, S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. in L Franco, C de Mulatier, M Paszynski, VV Krzhizhanovskaya, JJ Dongarra & PMA Sloot (eds), Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 14834 LNCS, Springer Science and Business Media Deutschland GmbH, pp. 222-230, 24th International Conference on Computational Science, ICCS 2024, Malaga, Spain, 07/2/24. https://doi.org/10.1007/978-3-031-63759-9_27

S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. / Shaik, Kareem; Wang, Dali; Zheng, Weijian et al.
Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings. ed. / Leonardo Franco; Clélia de Mulatier; Maciej Paszynski; Valeria V. Krzhizhanovskaya; Jack J. Dongarra; Peter M. A. Sloot. Springer Science and Business Media Deutschland GmbH, 2024. p. 222-230 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 14834 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - S3LLM

T2 - 24th International Conference on Computational Science, ICCS 2024

AU - Shaik, Kareem

AU - Wang, Dali

AU - Zheng, Weijian

AU - Cao, Qinglei

AU - Fan, Heng

AU - Schwartz, Peter

AU - Feng, Yunhe

N1 - Publisher Copyright: © The Author(s), under exclusive license to Springer Nature Switzerland AG 2024.

PY - 2024

Y1 - 2024

N2 - The understanding of large-scale scientific software is a significant challenge due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large language models (LLMs), provides novel pathways for understanding such complex scientific codes. This paper presents S3LLM, an LLM-based framework designed to enable the examination of source code, code metadata, and summarized information in conjunction with textual technical reports in an interactive, conversational manner through a user-friendly interface. S3LLM leverages open-source LLaMA-2 models to enhance code analysis through the automatic transformation of natural language queries into domain-specific language (DSL) queries. In addition, S3LLM is equipped to handle diverse metadata types, including DOT, SQL, and customized formats. Furthermore, S3LLM incorporates retrieval-augmented generation (RAG) and LangChain technologies to directly query extensive documents. S3LLM demonstrates the potential of using locally deployed open-source LLMs for the rapid understanding of large-scale scientific computing software, eliminating the need for extensive coding expertise and thereby making the process more efficient and effective. S3LLM is available at https://github.com/ResponsibleAILab/s3llm.

AB - The understanding of large-scale scientific software is a significant challenge due to its diverse codebase, extensive code length, and target computing architectures. The emergence of generative AI, specifically large language models (LLMs), provides novel pathways for understanding such complex scientific codes. This paper presents S3LLM, an LLM-based framework designed to enable the examination of source code, code metadata, and summarized information in conjunction with textual technical reports in an interactive, conversational manner through a user-friendly interface. S3LLM leverages open-source LLaMA-2 models to enhance code analysis through the automatic transformation of natural language queries into domain-specific language (DSL) queries. In addition, S3LLM is equipped to handle diverse metadata types, including DOT, SQL, and customized formats. Furthermore, S3LLM incorporates retrieval-augmented generation (RAG) and LangChain technologies to directly query extensive documents. S3LLM demonstrates the potential of using locally deployed open-source LLMs for the rapid understanding of large-scale scientific computing software, eliminating the need for extensive coding expertise and thereby making the process more efficient and effective. S3LLM is available at https://github.com/ResponsibleAILab/s3llm.

KW - ChatGPT

KW - E3SM Land Model

KW - Large-Scale Scientific Software

KW - LLaMA

KW - LLM

KW - Research Software Analysis

KW - Retrieval Augmented Generation (RAG)

UR - http://www.scopus.com/inward/record.url?scp=85199586809&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-63759-9_27

DO - 10.1007/978-3-031-63759-9_27

M3 - Conference contribution

AN - SCOPUS:85199586809

SN - 9783031637582

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 222

EP - 230

BT - Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings

A2 - Franco, Leonardo

A2 - de Mulatier, Clélia

A2 - Paszynski, Maciej

A2 - Krzhizhanovskaya, Valeria V.

A2 - Dongarra, Jack J.

A2 - Sloot, Peter M. A.

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 2 July 2024 through 4 July 2024

ER -

Shaik K, Wang D, Zheng W, Cao Q, Fan H, Schwartz P et al. S3LLM: Large-Scale Scientific Software Understanding with LLMs Using Source, Metadata, and Document. In Franco L, de Mulatier C, Paszynski M, Krzhizhanovskaya VV, Dongarra JJ, Sloot PMA, editors, Computational Science – ICCS 2024 - 24th International Conference, 2024, Proceedings. Springer Science and Business Media Deutschland GmbH. 2024. p. 222-230. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-031-63759-9_27