User-based I/O Profiling for Leadership Scale HPC Workloads

Ahmad Hossein Yazdani; Arnab K. Paul; Ahmad Maroof Karimi; Feiyi Wang; Ali Butt

doi:10.1145/3700838.3700865

User-based I/O Profiling for Leadership Scale HPC Workloads

Ahmad Hossein Yazdani, Arnab K. Paul, Ahmad Maroof Karimi, Feiyi Wang, Ali Butt

Analytics and AI Methods at Scale

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

I/O constitutes a significant portion of most of the application run-time. Spawning many such applications concurrently on an HPC system leads to severe I/O contention. Thus, understanding and subsequently reducing I/O contention induced by such multi-tenancy is critical for the efficient and reliable performance of the HPC system. In this study, we demonstrate that an application's performance is influenced by the command line arguments passed to the job submission. We model an application's I/O behavior based on two factors: past I/O behavior within a time window and user-configured I/O settings via command-line arguments. We conclude that I/O patterns for well-known HPC applications like E3SM and LAMMP are predictable, with an average uncertainty below 0.25 (A probability of 80%) and near zero (A probability of 100%) within a day. However, I/O pattern variance increases as the study time window lengthens. Additionally, we show that for 38 users and at least 50 applications constituting approximately 93000 job submissions, there is a high correlation between a submitted command line and the past command lines made within 1 to 10 days submitted by the user. We claim the length of this time window is unique per user.

Original language	English
Title of host publication	ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking
Publisher	Association for Computing Machinery, Inc
Pages	181-190
Number of pages	10
ISBN (Electronic)	9798400710629
DOIs	https://doi.org/10.1145/3700838.3700865
State	Published - Jan 4 2025
Event	26th International Conference on Distributed Computing and Networking, ICDCN 2025 - Hyderabad, India Duration: Jan 4 2025 → Jan 7 2025

Publication series

Name	ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking

Conference

Conference	26th International Conference on Distributed Computing and Networking, ICDCN 2025
Country/Territory	India
City	Hyderabad
Period	01/4/25 → 01/7/25

Funding

We thank our anonymous reviewers for their detailed feedback and valuable suggestions. This work is sponsored in part by the NSF under the grants: CSR-2106634, CCF-1919113/1919075, CNS2045680, OAC-2004751, and OAC-2106446, Office of Science of the U.S. Department of Energy under the grant DE-AC05-00OR22725, SERB, Govt. of India Start-up Research grant SRG/2023/002445, BITS CRDF under grant C1/23/173 and BITS Pilani under the grants: BBF/BITS(G)/FY2022-23/BCPS-123/24-25/R1 and GOA/ACG/2022- 2023/Oct/11. Results presented in this paper were obtained using the OLCF at Oak Ridge National Laboratory.

Keywords

Darshan
High Performance Computing
I/O characterization
I/O profiling
I/O scheduler

Access to Document

10.1145/3700838.3700865

Cite this

Yazdani, A. H., Paul, A. K., Karimi, A. M., Wang, F., & Butt, A. (2025). User-based I/O Profiling for Leadership Scale HPC Workloads. In ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking (pp. 181-190). (ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking). Association for Computing Machinery, Inc. https://doi.org/10.1145/3700838.3700865

Yazdani, Ahmad Hossein ; Paul, Arnab K. ; Karimi, Ahmad Maroof et al. / User-based I/O Profiling for Leadership Scale HPC Workloads. ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking. Association for Computing Machinery, Inc, 2025. pp. 181-190 (ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking).

@inproceedings{f458ca09d996438abf9c6b6236dcfa12,

title = "User-based I/O Profiling for Leadership Scale HPC Workloads",

abstract = "I/O constitutes a significant portion of most of the application run-time. Spawning many such applications concurrently on an HPC system leads to severe I/O contention. Thus, understanding and subsequently reducing I/O contention induced by such multi-tenancy is critical for the efficient and reliable performance of the HPC system. In this study, we demonstrate that an application's performance is influenced by the command line arguments passed to the job submission. We model an application's I/O behavior based on two factors: past I/O behavior within a time window and user-configured I/O settings via command-line arguments. We conclude that I/O patterns for well-known HPC applications like E3SM and LAMMP are predictable, with an average uncertainty below 0.25 (A probability of 80%) and near zero (A probability of 100%) within a day. However, I/O pattern variance increases as the study time window lengthens. Additionally, we show that for 38 users and at least 50 applications constituting approximately 93000 job submissions, there is a high correlation between a submitted command line and the past command lines made within 1 to 10 days submitted by the user. We claim the length of this time window is unique per user.",

keywords = "Darshan, High Performance Computing, I/O characterization, I/O profiling, I/O scheduler",

author = "Yazdani, {Ahmad Hossein} and Paul, {Arnab K.} and Karimi, {Ahmad Maroof} and Feiyi Wang and Ali Butt",

note = "Publisher Copyright: {\textcopyright} 2025 Copyright held by the owner/author(s).; 26th International Conference on Distributed Computing and Networking, ICDCN 2025 ; Conference date: 04-01-2025 Through 07-01-2025",

year = "2025",

month = jan,

day = "4",

doi = "10.1145/3700838.3700865",

language = "English",

series = "ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking",

publisher = "Association for Computing Machinery, Inc",

pages = "181--190",

booktitle = "ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking",

}

Yazdani, AH, Paul, AK, Karimi, AM , Wang, F & Butt, A 2025, User-based I/O Profiling for Leadership Scale HPC Workloads. in ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking. ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking, Association for Computing Machinery, Inc, pp. 181-190, 26th International Conference on Distributed Computing and Networking, ICDCN 2025, Hyderabad, India, 01/4/25. https://doi.org/10.1145/3700838.3700865

User-based I/O Profiling for Leadership Scale HPC Workloads. / Yazdani, Ahmad Hossein; Paul, Arnab K.; Karimi, Ahmad Maroof et al.
ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking. Association for Computing Machinery, Inc, 2025. p. 181-190 (ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - User-based I/O Profiling for Leadership Scale HPC Workloads

AU - Yazdani, Ahmad Hossein

AU - Paul, Arnab K.

AU - Karimi, Ahmad Maroof

AU - Wang, Feiyi

AU - Butt, Ali

PY - 2025/1/4

Y1 - 2025/1/4

N2 - I/O constitutes a significant portion of most of the application run-time. Spawning many such applications concurrently on an HPC system leads to severe I/O contention. Thus, understanding and subsequently reducing I/O contention induced by such multi-tenancy is critical for the efficient and reliable performance of the HPC system. In this study, we demonstrate that an application's performance is influenced by the command line arguments passed to the job submission. We model an application's I/O behavior based on two factors: past I/O behavior within a time window and user-configured I/O settings via command-line arguments. We conclude that I/O patterns for well-known HPC applications like E3SM and LAMMP are predictable, with an average uncertainty below 0.25 (A probability of 80%) and near zero (A probability of 100%) within a day. However, I/O pattern variance increases as the study time window lengthens. Additionally, we show that for 38 users and at least 50 applications constituting approximately 93000 job submissions, there is a high correlation between a submitted command line and the past command lines made within 1 to 10 days submitted by the user. We claim the length of this time window is unique per user.

AB - I/O constitutes a significant portion of most of the application run-time. Spawning many such applications concurrently on an HPC system leads to severe I/O contention. Thus, understanding and subsequently reducing I/O contention induced by such multi-tenancy is critical for the efficient and reliable performance of the HPC system. In this study, we demonstrate that an application's performance is influenced by the command line arguments passed to the job submission. We model an application's I/O behavior based on two factors: past I/O behavior within a time window and user-configured I/O settings via command-line arguments. We conclude that I/O patterns for well-known HPC applications like E3SM and LAMMP are predictable, with an average uncertainty below 0.25 (A probability of 80%) and near zero (A probability of 100%) within a day. However, I/O pattern variance increases as the study time window lengthens. Additionally, we show that for 38 users and at least 50 applications constituting approximately 93000 job submissions, there is a high correlation between a submitted command line and the past command lines made within 1 to 10 days submitted by the user. We claim the length of this time window is unique per user.

KW - Darshan

KW - High Performance Computing

KW - I/O characterization

KW - I/O profiling

KW - I/O scheduler

UR - http://www.scopus.com/inward/record.url?scp=85218353853&partnerID=8YFLogxK

U2 - 10.1145/3700838.3700865

DO - 10.1145/3700838.3700865

M3 - Conference contribution

AN - SCOPUS:85218353853

T3 - ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking

SP - 181

EP - 190

BT - ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking

PB - Association for Computing Machinery, Inc

T2 - 26th International Conference on Distributed Computing and Networking, ICDCN 2025

Y2 - 4 January 2025 through 7 January 2025

ER -

Yazdani AH, Paul AK, Karimi AM , Wang F, Butt A. User-based I/O Profiling for Leadership Scale HPC Workloads. In ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking. Association for Computing Machinery, Inc. 2025. p. 181-190. (ICDCN 2025 - Proceedings of the 26th International Conference on Distributed Computing and Networking). doi: 10.1145/3700838.3700865

User-based I/O Profiling for Leadership Scale HPC Workloads

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this