Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model

Tijmen de Haan; Yuan Sen Ting; Tirthankar Ghosal; Tuan Dung Nguyen; Alberto Accomazzi; Azton Wells; Nesar Ramachandra; Rui Pan; Zechang Sun

doi:10.1038/s41598-025-97131-y

Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model

Tijmen de Haan, Yuan Sen Ting, Tirthankar Ghosal, Tuan Dung Nguyen, Alberto Accomazzi, Azton Wells, Nesar Ramachandra, Rui Pan, Zechang Sun

Analytics and AI Methods at Scale

Research output: Contribution to journal › Article › peer-review

2 Scopus citations

Abstract

AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9% on the AstroMLab-1 benchmark, greatly outperforming all models—proprietary and open-weight—in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.

Original language	English
Article number	13751
Journal	Scientific Reports
Volume	15
Issue number	1
DOIs	https://doi.org/10.1038/s41598-025-97131-y
State	Published - Dec 2025

Funding

This research used resources of the Oak Ridge Leadership Computing Facility (OLCF), which is a DOE Office of Science User Facility at the Oak Ridge National Laboratory supported by the U.S. Department of Energy under Contract No. DE-AC05-00OR22725 and support from Microsoft’s Accelerating Foundation Models Research (AFMR) program. TdH was supported by World Premier International Research Center Initiative (WPI), MEXT, Japan. YST is supported by the National Science Foundation under Grant No. 2406729. Work at Argonne National Lab is supported by UChicago Argonne LLC, Operator of Argonne National Laboratory. Argonne, a U.S. Department of Energy Office of Science Laboratory, is operated under contract no. DE-AC02-06CH11357. A special thanks goes out to Cassie Reuter and Joshua Montgomery for acting as independent evaluators.

Keywords

AI assistant
Continued pretraining
Large-language model
Supervised fine-tuning

Access to Document

10.1038/s41598-025-97131-y

Cite this

@article{b76cc217e0244e8c903a30276062dc9b,

title = "Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model",

abstract = "AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9\% on the AstroMLab-1 benchmark, greatly outperforming all models—proprietary and open-weight—in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.",

keywords = "AI assistant, Continued pretraining, Large-language model, Supervised fine-tuning",

author = "\{de Haan\}, Tijmen and Ting, \{Yuan Sen\} and Tirthankar Ghosal and Nguyen, \{Tuan Dung\} and Alberto Accomazzi and Azton Wells and Nesar Ramachandra and Rui Pan and Zechang Sun",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.",

year = "2025",

month = dec,

doi = "10.1038/s41598-025-97131-y",

language = "English",

volume = "15",

journal = "Scientific Reports",

issn = "2045-2322",

publisher = "Nature Research",

number = "1",

}

TY - JOUR

T1 - Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model

AU - de Haan, Tijmen

AU - Ting, Yuan Sen

AU - Ghosal, Tirthankar

AU - Nguyen, Tuan Dung

AU - Accomazzi, Alberto

AU - Wells, Azton

AU - Ramachandra, Nesar

AU - Pan, Rui

AU - Sun, Zechang

N1 - Publisher Copyright: © The Author(s) 2025.

PY - 2025/12

Y1 - 2025/12

N2 - AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9% on the AstroMLab-1 benchmark, greatly outperforming all models—proprietary and open-weight—in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.

AB - AstroSage-Llama-3.1-8B is a domain-specialized natural-language AI assistant tailored for research in astronomy, astrophysics, cosmology, and astronomical instrumentation. Trained on the complete collection of astronomy-related arXiv papers from 2007 to 2024 along with millions of synthetically-generated question-answer pairs and other astronomical literature, AstroSage-Llama-3.1-8B demonstrates remarkable proficiency on a wide range of questions. AstroSage-Llama-3.1-8B scores 80.9% on the AstroMLab-1 benchmark, greatly outperforming all models—proprietary and open-weight—in the 8-billion parameter class, and performing on par with GPT-4o. This achievement demonstrates the potential of domain specialization in AI, suggesting that focused training can yield capabilities exceeding those of much larger, general-purpose models. AstroSage-Llama-3.1-8B is freely available, enabling widespread access to advanced AI capabilities for astronomical education and research.

KW - AI assistant

KW - Continued pretraining

KW - Large-language model

KW - Supervised fine-tuning

UR - https://www.scopus.com/pages/publications/105003250699

U2 - 10.1038/s41598-025-97131-y

DO - 10.1038/s41598-025-97131-y

M3 - Article

C2 - 40258872

AN - SCOPUS:105003250699

SN - 2045-2322

VL - 15

JO - Scientific Reports

JF - Scientific Reports

IS - 1

M1 - 13751

ER -

Achieving GPT-4o level performance in astronomy with a specialized 8B-parameter large language model

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this