Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O

Ranjan Sarpangala Venkatesh, Greg Eisenhauer, Norbert Podhorszki, Dmitry Ganyushin, Scott Klasky, Ada Gavrilovska

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

In HPC I/O middleware like the Adaptable I/O System (ADIOS) often mediates data transfers between applications. The metadata I/O generated by such systems often presents significant scaling and performance limitations. This work seeks improvement opportunities for metadata I/O by leveraging the DAOS storage systems, a recent storage system solution deployed on high-end systems such as the Aurora supercomputer. We investigate the tradeoffs and the design space for integrating I/O engines for the ADIOS middleware based on the different storage mechanisms supported by DAOS. We present a new DAOS-Array-ChunkSize-aligned engine which provides up to 2.3× improved performance than when using the existing DAOS-POSIX interface, without requiring any application modifications.

Original languageEnglish
Title of host publicationResearch Paper Proceedings of the ISC High Performance 2024
PublisherInstitute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)9783982633602
StatePublished - 2024
Event39th International Conference on High Performance Computing, ISC High Performance 2024 - Hamburg, Germany
Duration: May 12 2024May 16 2024

Publication series

NameResearch Paper Proceedings of the ISC High Performance 2024

Conference

Conference39th International Conference on High Performance Computing, ISC High Performance 2024
Country/TerritoryGermany
CityHamburg
Period05/12/2405/16/24

Keywords

  • ADIOS
  • DAOS
  • data management
  • HPC
  • metadata
  • object storage
  • workflow I/O

Fingerprint

Dive into the research topics of 'Optimizing Metadata Exchange: Leveraging DAOS for ADIOS Metadata I/O'. Together they form a unique fingerprint.

Cite this