Abstract
In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.
| Original language | English |
|---|---|
| Title of host publication | Research Paper Proceedings of the ISC High Performance 2024 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9783982633602 |
| State | Published - 2024 |
| Event | 39th International Conference on High Performance Computing, ISC High Performance 2024 - Hamburg, Germany Duration: May 12 2024 → May 16 2024 |
Publication series
| Name | Research Paper Proceedings of the ISC High Performance 2024 |
|---|
Conference
| Conference | 39th International Conference on High Performance Computing, ISC High Performance 2024 |
|---|---|
| Country/Territory | Germany |
| City | Hamburg |
| Period | 05/12/24 → 05/16/24 |
Funding
This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) and the Cambridge Open Zettascale Laboratory. These facilities are operated by the University of Cambridge Research Computing Service (www.csd3.cam.ac.uk), using resources provided by Dell EMC and Intel, Tier-2 funding from the Engineering and Physical Sciences Research Council (capital grant EP/T022159/1), The Excalibur Project (https://www.scd.stfc.ac.uk/Pages/Excalibur-Programme.aspx), and DiRAC (www.dirac.ac.uk). Funding from the latter two is provided by The Science and Technology Facilities Council.
Keywords
- ADIOS
- DAOS
- HPC
- data management
- metadata
- object storage
- workflow I/O
Fingerprint
Dive into the research topics of 'Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O'. Together they form a unique fingerprint.Cite this
- APA
- Author
- BIBTEX
- Harvard
- Standard
- RIS
- Vancouver