Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context

Ranjan Sarpangala Venkatesh; Greg Eisenhauer; Scott Klasky; Ada Gavrilovska

doi:10.1145/3624062.3624193

Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context

Ranjan Sarpangala Venkatesh, Greg Eisenhauer, Scott Klasky, Ada Gavrilovska

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.

Original language	English
Title of host publication	Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Publisher	Association for Computing Machinery
Pages	1223-1228
Number of pages	6
ISBN (Electronic)	9798400707858
DOIs	https://doi.org/10.1145/3624062.3624193
State	Published - Nov 12 2023
Event	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 - Denver, United States Duration: Nov 12 2023 → Nov 17 2023

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Country/Territory	United States
City	Denver
Period	11/12/23 → 11/17/23

Funding

This research was funded by the Exascale Computing Project (17-SC-20-SC) and with partial support from the SRC/DARRA JUMP2.0 center PRISM. This work was performed using resources provided by the Cambridge Service for Data Driven Discovery (CSD3) and the Cambridge Open Zettascale Laboratory, funded partially by The Science and Technology Facilities Council.

Funders	Funder number
Science and Technology Facilities Council

Keywords

ADIOS
DAOS
Data management
HPC metadata
I/O middleware

Access to Document

10.1145/3624062.3624193

Cite this

Sarpangala Venkatesh, R., Eisenhauer, G., Klasky, S., & Gavrilovska, A. (2023). Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 (pp. 1223-1228). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3624062.3624193

Sarpangala Venkatesh, Ranjan ; Eisenhauer, Greg ; Klasky, Scott et al. / Enhancing Metadata Transfer Efficiency : Unlocking the Potential of DAOS in the ADIOS context. Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. pp. 1223-1228 (ACM International Conference Proceeding Series).

@inproceedings{b67ab092a321469db8f5a7b4bef144b2,

title = "Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context",

abstract = "In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.",

keywords = "ADIOS, DAOS, Data management, HPC metadata, I/O middleware",

author = "{Sarpangala Venkatesh}, Ranjan and Greg Eisenhauer and Scott Klasky and Ada Gavrilovska",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 ; Conference date: 12-11-2023 Through 17-11-2023",

year = "2023",

month = nov,

day = "12",

doi = "10.1145/3624062.3624193",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "1223--1228",

booktitle = "Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023",

}

Sarpangala Venkatesh, R, Eisenhauer, G, Klasky, S & Gavrilovska, A 2023, Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context. in Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 1223-1228, 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023, Denver, United States, 11/12/23. https://doi.org/10.1145/3624062.3624193

Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context. / Sarpangala Venkatesh, Ranjan; Eisenhauer, Greg; Klasky, Scott et al.
Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. p. 1223-1228 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Enhancing Metadata Transfer Efficiency

T2 - 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

AU - Sarpangala Venkatesh, Ranjan

AU - Eisenhauer, Greg

AU - Klasky, Scott

AU - Gavrilovska, Ada

PY - 2023/11/12

Y1 - 2023/11/12

N2 - In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.

AB - In HPC, data movement between applications is typically facilitated by I/O middleware, such as the Adaptable I/O System (ADIOS). This middleware leverages the capabilities of the underlying storage services, to facilitate data movement and distribution. A recent storage system, Intel DAOS, promises to deliver new capabilities for achieving performance and scalability on emerging memory/storage systems. DAOS has already deployed in University of Cambridge, TACC, and the upcoming Aurora supercomputer. This paper investigates the performance tradeoffs associated with mapping ADIOS over one of the many different DAOS interfaces and data models, and makes recommendations for their efficient use.

KW - ADIOS

KW - DAOS

KW - Data management

KW - HPC metadata

KW - I/O middleware

UR - http://www.scopus.com/inward/record.url?scp=85178133456&partnerID=8YFLogxK

U2 - 10.1145/3624062.3624193

DO - 10.1145/3624062.3624193

M3 - Conference contribution

AN - SCOPUS:85178133456

T3 - ACM International Conference Proceeding Series

SP - 1223

EP - 1228

BT - Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

PB - Association for Computing Machinery

Y2 - 12 November 2023 through 17 November 2023

ER -

Sarpangala Venkatesh R, Eisenhauer G, Klasky S, Gavrilovska A. Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery. 2023. p. 1223-1228. (ACM International Conference Proceeding Series). doi: 10.1145/3624062.3624193

Enhancing Metadata Transfer Efficiency: Unlocking the Potential of DAOS in the ADIOS context

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this