TY - GEN
T1 - Enabling High- Throughput Parallel I/O in Particle-in-Cell Monte Carlo Simulations with openPMD and Darshan I/O Monitoring
AU - Williams, Jeremy J.
AU - Medeiros, Daniel
AU - Costea, Stefan
AU - Tskhakaya, David
AU - Poeschel, Franz
AU - Widera, Rene
AU - Huebl, Axel
AU - Klasky, Scott
AU - Podhorszki, Norbert
AU - Kos, Leon
AU - Podolnik, Ales
AU - Hromadka, Jakub
AU - Narwal, Tapish
AU - Steiniger, Klaus
AU - Bussmann, Michael
AU - Laure, Erwin
AU - Markidis, Stefano
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - Large-scale HPC simulations of plasma dynamics in fusion devices require efficient parallel I/O to avoid slowing down the simulation and to enable the post-processing of critical information. Such complex simulations lacking parallel I/O capabilities may encounter performance bottlenecks, hindering their effectiveness in data-intensive computing tasks. In this work, we focus on introducing and enhancing the efficiency of parallel I/O operations in Particle-in-Cell Monte Carlo simu-lations. We first evaluate the scalability of BIT1, a massively-parallel electrostatic PIC MC code, determining its initial write throughput capabilities and performance bottlenecks using an HPC I/O performance monitoring tool, Darshan. We design and develop an adaptor to the openPMD I/O interface that allows us to stream PIC particle and field information to I/O using the BP4 backend, aggressively optimized for I/O efficiency, including the highly efficient ADIOS2 interface. Next, we explore advanced optimization techniques such as data compression, aggregation, and Lustre file striping, achieving write throughput improvements while enhancing data storage efficiency. Finally, we analyze the enhanced high-throughput parallel I/O and storage capabilities achieved through the integration of openPMD with rapid metadata extraction in BP4 format. Our study demonstrates that the integration of openPMD and advanced I/O optimizations significantly enhances BIT1's I/O performance and storage capabilities, successfully introducing high throughput parallel I/O and surpassing the capabilities of traditional file I/O.
AB - Large-scale HPC simulations of plasma dynamics in fusion devices require efficient parallel I/O to avoid slowing down the simulation and to enable the post-processing of critical information. Such complex simulations lacking parallel I/O capabilities may encounter performance bottlenecks, hindering their effectiveness in data-intensive computing tasks. In this work, we focus on introducing and enhancing the efficiency of parallel I/O operations in Particle-in-Cell Monte Carlo simu-lations. We first evaluate the scalability of BIT1, a massively-parallel electrostatic PIC MC code, determining its initial write throughput capabilities and performance bottlenecks using an HPC I/O performance monitoring tool, Darshan. We design and develop an adaptor to the openPMD I/O interface that allows us to stream PIC particle and field information to I/O using the BP4 backend, aggressively optimized for I/O efficiency, including the highly efficient ADIOS2 interface. Next, we explore advanced optimization techniques such as data compression, aggregation, and Lustre file striping, achieving write throughput improvements while enhancing data storage efficiency. Finally, we analyze the enhanced high-throughput parallel I/O and storage capabilities achieved through the integration of openPMD with rapid metadata extraction in BP4 format. Our study demonstrates that the integration of openPMD and advanced I/O optimizations significantly enhances BIT1's I/O performance and storage capabilities, successfully introducing high throughput parallel I/O and surpassing the capabilities of traditional file I/O.
KW - ADIOS2
KW - Darshan
KW - Distributed Storage
KW - Efficient Data Processing
KW - Large-Scale PIC Simulations
KW - openPMD
KW - Parallel I/O
UR - http://www.scopus.com/inward/record.url?scp=85204321286&partnerID=8YFLogxK
U2 - 10.1109/CLUSTERWorkshops61563.2024.00022
DO - 10.1109/CLUSTERWorkshops61563.2024.00022
M3 - Conference contribution
AN - SCOPUS:85204321286
T3 - Proceedings - 2024 IEEE International Conference on Cluster Computing Workshops, CLUSTER Workshops 2024
SP - 86
EP - 95
BT - Proceedings - 2024 IEEE International Conference on Cluster Computing Workshops, CLUSTER Workshops 2024
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2024 IEEE International Conference on Cluster Computing Workshops, CLUSTER Workshops 2024
Y2 - 24 September 2024 through 27 September 2024
ER -