PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory

Luke Logan; Jay Lofstead; Scott Levy; Patrick Widener; Xian He Sun; Anthony Kougkas

doi:10.1109/Cluster48925.2021.00098

PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory

Luke Logan, Jay Lofstead, Scott Levy, Patrick Widener, Xian He Sun, Anthony Kougkas

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

Persistent memory (PMEM) devices can achieve comparable performance to DRAM while providing significantly more capacity. This has made the technology compelling as an expansion to main memory. Rethinking PMEM as storage devices can offer a high performance buffering layer for HPC applications to temporarily, but safely store data. However, modern parallel I/O libraries, such as HDF5 and pNetCDF, are complicated and introduce significant software and metadata overheads when persisting data to these storage devices, wasting much of their potential. In this work, we explore the potential of PMEM as storage through pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory. We demonstrate that our approach is up to 2x faster than other popular parallel I/O libraries under real workloads.

Original language	English
Title of host publication	Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	664-670
Number of pages	7
ISBN (Electronic)	9781728196664
DOIs	https://doi.org/10.1109/Cluster48925.2021.00098
State	Published - 2021
Externally published	Yes
Event	2021 IEEE International Conference on Cluster Computing, Cluster 2021 - Virtual, Portland, United States Duration: Sep 7 2021 → Sep 10 2021

Publication series

Name	Proceedings - IEEE International Conference on Cluster Computing, ICCC
Volume	2021-September
ISSN (Print)	1552-5244

Conference

Conference	2021 IEEE International Conference on Cluster Computing, Cluster 2021
Country/Territory	United States
City	Virtual, Portland
Period	09/7/21 → 09/10/21

Funding

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Keywords

Data Layout
Libraries
Memory Management
Memory Mapped I/O
Parallel I/O
Persistent Memory
Storage Management

Access to Document

10.1109/Cluster48925.2021.00098

Cite this

Logan, L., Lofstead, J., Levy, S., Widener, P., Sun, X. H., & Kougkas, A. (2021). PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory. In Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021 (pp. 664-670). (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2021-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/Cluster48925.2021.00098

Logan, Luke ; Lofstead, Jay ; Levy, Scott et al. / PMEMCPY : A simple, lightweight, and portable I/O library for storing data in persistent memory. Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 664-670 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).

@inproceedings{28465bf96337442d8a2add4279a94d96,

title = "PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory",

abstract = "Persistent memory (PMEM) devices can achieve comparable performance to DRAM while providing significantly more capacity. This has made the technology compelling as an expansion to main memory. Rethinking PMEM as storage devices can offer a high performance buffering layer for HPC applications to temporarily, but safely store data. However, modern parallel I/O libraries, such as HDF5 and pNetCDF, are complicated and introduce significant software and metadata overheads when persisting data to these storage devices, wasting much of their potential. In this work, we explore the potential of PMEM as storage through pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory. We demonstrate that our approach is up to 2x faster than other popular parallel I/O libraries under real workloads.",

keywords = "Data Layout, Libraries, Memory Management, Memory Mapped I/O, Parallel I/O, Persistent Memory, Storage Management",

author = "Luke Logan and Jay Lofstead and Scott Levy and Patrick Widener and Sun, {Xian He} and Anthony Kougkas",

note = "Publisher Copyright: {\textcopyright}2021 IEEE.; 2021 IEEE International Conference on Cluster Computing, Cluster 2021 ; Conference date: 07-09-2021 Through 10-09-2021",

year = "2021",

doi = "10.1109/Cluster48925.2021.00098",

language = "English",

series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "664--670",

booktitle = "Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021",

}

Logan, L, Lofstead, J, Levy, S, Widener, P, Sun, XH & Kougkas, A 2021, PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory. in Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021. Proceedings - IEEE International Conference on Cluster Computing, ICCC, vol. 2021-September, Institute of Electrical and Electronics Engineers Inc., pp. 664-670, 2021 IEEE International Conference on Cluster Computing, Cluster 2021, Virtual, Portland, United States, 09/7/21. https://doi.org/10.1109/Cluster48925.2021.00098

PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory. / Logan, Luke; Lofstead, Jay; Levy, Scott et al.
Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 664-670 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2021-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - PMEMCPY

T2 - 2021 IEEE International Conference on Cluster Computing, Cluster 2021

AU - Logan, Luke

AU - Lofstead, Jay

AU - Levy, Scott

AU - Widener, Patrick

AU - Sun, Xian He

AU - Kougkas, Anthony

PY - 2021

Y1 - 2021

N2 - Persistent memory (PMEM) devices can achieve comparable performance to DRAM while providing significantly more capacity. This has made the technology compelling as an expansion to main memory. Rethinking PMEM as storage devices can offer a high performance buffering layer for HPC applications to temporarily, but safely store data. However, modern parallel I/O libraries, such as HDF5 and pNetCDF, are complicated and introduce significant software and metadata overheads when persisting data to these storage devices, wasting much of their potential. In this work, we explore the potential of PMEM as storage through pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory. We demonstrate that our approach is up to 2x faster than other popular parallel I/O libraries under real workloads.

AB - Persistent memory (PMEM) devices can achieve comparable performance to DRAM while providing significantly more capacity. This has made the technology compelling as an expansion to main memory. Rethinking PMEM as storage devices can offer a high performance buffering layer for HPC applications to temporarily, but safely store data. However, modern parallel I/O libraries, such as HDF5 and pNetCDF, are complicated and introduce significant software and metadata overheads when persisting data to these storage devices, wasting much of their potential. In this work, we explore the potential of PMEM as storage through pMEMCPY: a simple, lightweight, and portable I/O library for storing data in persistent memory. We demonstrate that our approach is up to 2x faster than other popular parallel I/O libraries under real workloads.

KW - Data Layout

KW - Libraries

KW - Memory Management

KW - Memory Mapped I/O

KW - Parallel I/O

KW - Persistent Memory

KW - Storage Management

UR - http://www.scopus.com/inward/record.url?scp=85125994131&partnerID=8YFLogxK

U2 - 10.1109/Cluster48925.2021.00098

DO - 10.1109/Cluster48925.2021.00098

M3 - Conference contribution

AN - SCOPUS:85125994131

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

SP - 664

EP - 670

BT - Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 7 September 2021 through 10 September 2021

ER -

Logan L, Lofstead J, Levy S, Widener P, Sun XH, Kougkas A. PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory. In Proceedings - 2021 IEEE International Conference on Cluster Computing, Cluster 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 664-670. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). doi: 10.1109/Cluster48925.2021.00098

PMEMCPY: A simple, lightweight, and portable I/O library for storing data in persistent memory

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this