neCODEC: Nearline data compression for scientific applications

Yuan Tian; Cong Xu; Weikuan Yu; Jeffrey S. Vetter; Scott Klasky; Honggao Liu; Saad Biaz

doi:10.1007/s10586-013-0265-8

neCODEC: Nearline data compression for scientific applications

Yuan Tian, Cong Xu, Weikuan Yu, Jeffrey S. Vetter, Scott Klasky, Honggao Liu, Saad Biaz

Research output: Contribution to journal › Article › peer-review

Abstract

Advances on multicore technologies lead to processors with tens and soon hundreds of cores in a single socket, resulting in an ever growing gap between computing power and available memory and I/O bandwidths for data handling. It would be beneficial if some of the computing power can be transformed into gains of I/O efficiency, thereby reducing this speed disparity between computing and I/O. In this paper, we design and implement a NEarline data COmpression and DECompression (neCODEC) scheme for data-intensive parallel applications. Several salient techniques are introduced in neCODEC, including asynchronous compression threads, elastic file representation, distributed metadata handling, and balanced subfile distribution. Our performance evaluation indicates that neCODEC can improve the performance of a variety of data-intensive microbenchmarks and scientific applications. Particularly, neCODEC is capable of increasing the effective bandwidth of S3D, a combustion simulation code, by more than 5 times.

Original language	English
Pages (from-to)	475-486
Number of pages	12
Journal	Cluster Computing
Volume	17
Issue number	2
DOIs	https://doi.org/10.1007/s10586-013-0265-8
State	Published - Jan 2014

Funding

Acknowledgements This work is funded in part by National Science Foundation awards CNS-0917137 and CNS-1059376. This research is sponsored in part by the Office of Advanced Scientific Computing Research; U.S. Department of Energy. This research is conducted with high performance computational resources provided by the Louisiana Optical Network Initiative (http://www.loni.org). We are very grateful for the technical support from the LONI team.

Funders	Funder number
National Science Foundation	1059376, CNS-1059376, 1320016, CNS-0917137
U.S. Department of Energy
Advanced Scientific Computing Research

Keywords

Data compression
Lustre
MPI-IO

Access to Document

10.1007/s10586-013-0265-8

Cite this

@article{a0c0397082c7484489dcd0208611320c,

title = "neCODEC: Nearline data compression for scientific applications",

abstract = "Advances on multicore technologies lead to processors with tens and soon hundreds of cores in a single socket, resulting in an ever growing gap between computing power and available memory and I/O bandwidths for data handling. It would be beneficial if some of the computing power can be transformed into gains of I/O efficiency, thereby reducing this speed disparity between computing and I/O. In this paper, we design and implement a NEarline data COmpression and DECompression (neCODEC) scheme for data-intensive parallel applications. Several salient techniques are introduced in neCODEC, including asynchronous compression threads, elastic file representation, distributed metadata handling, and balanced subfile distribution. Our performance evaluation indicates that neCODEC can improve the performance of a variety of data-intensive microbenchmarks and scientific applications. Particularly, neCODEC is capable of increasing the effective bandwidth of S3D, a combustion simulation code, by more than 5 times.",

keywords = "Data compression, Lustre, MPI-IO",

author = "Yuan Tian and Cong Xu and Weikuan Yu and Vetter, {Jeffrey S.} and Scott Klasky and Honggao Liu and Saad Biaz",

year = "2014",

month = jan,

doi = "10.1007/s10586-013-0265-8",

language = "English",

volume = "17",

pages = "475--486",

journal = "Cluster Computing",

issn = "1386-7857",

publisher = "Springer",

number = "2",

}

TY - JOUR

T1 - neCODEC

T2 - Nearline data compression for scientific applications

AU - Tian, Yuan

AU - Xu, Cong

AU - Yu, Weikuan

AU - Vetter, Jeffrey S.

AU - Klasky, Scott

AU - Liu, Honggao

AU - Biaz, Saad

PY - 2014/1

Y1 - 2014/1

N2 - Advances on multicore technologies lead to processors with tens and soon hundreds of cores in a single socket, resulting in an ever growing gap between computing power and available memory and I/O bandwidths for data handling. It would be beneficial if some of the computing power can be transformed into gains of I/O efficiency, thereby reducing this speed disparity between computing and I/O. In this paper, we design and implement a NEarline data COmpression and DECompression (neCODEC) scheme for data-intensive parallel applications. Several salient techniques are introduced in neCODEC, including asynchronous compression threads, elastic file representation, distributed metadata handling, and balanced subfile distribution. Our performance evaluation indicates that neCODEC can improve the performance of a variety of data-intensive microbenchmarks and scientific applications. Particularly, neCODEC is capable of increasing the effective bandwidth of S3D, a combustion simulation code, by more than 5 times.

AB - Advances on multicore technologies lead to processors with tens and soon hundreds of cores in a single socket, resulting in an ever growing gap between computing power and available memory and I/O bandwidths for data handling. It would be beneficial if some of the computing power can be transformed into gains of I/O efficiency, thereby reducing this speed disparity between computing and I/O. In this paper, we design and implement a NEarline data COmpression and DECompression (neCODEC) scheme for data-intensive parallel applications. Several salient techniques are introduced in neCODEC, including asynchronous compression threads, elastic file representation, distributed metadata handling, and balanced subfile distribution. Our performance evaluation indicates that neCODEC can improve the performance of a variety of data-intensive microbenchmarks and scientific applications. Particularly, neCODEC is capable of increasing the effective bandwidth of S3D, a combustion simulation code, by more than 5 times.

KW - Data compression

KW - Lustre

KW - MPI-IO

UR - http://www.scopus.com/inward/record.url?scp=84901595912&partnerID=8YFLogxK

U2 - 10.1007/s10586-013-0265-8

DO - 10.1007/s10586-013-0265-8

M3 - Article

AN - SCOPUS:84901595912

SN - 1386-7857

VL - 17

SP - 475

EP - 486

JO - Cluster Computing

JF - Cluster Computing

IS - 2

ER -

neCODEC: Nearline data compression for scientific applications

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this