Fast Algorithms for Scientific Data Compression

Tania Banerjee, Jaemoon Lee, Jong Choi, Qian Gong, Jieyang Chen, Scott Klasky, Anand Rangarajan, Sanjay Ranka

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Many scientific simulations and experiments generate terabytes to petabytes of data daily, necessitating data compression techniques. Unlike video and image compression, scientists require methods that accurately preserve primary data (PD) and derived quantities of interest (QoIs). In our previous work, we demonstrated the effectiveness of hybrid compression techniques that combine machine learning with traditional approaches. This paper presents innovative computational techniques aimed at expediting the compression pipeline. Our experiments, conducted on two distinct platforms with a large-scale XGC-based fusion simulation, demonstrate that the overhead incurred by these new approaches is less than one percent of the computational resources needed for the simulation.

Original languageEnglish
Title of host publicationProceedings - 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics, HiPC 2023
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages143-152
Number of pages10
ISBN (Electronic)9798350383225
DOIs
StatePublished - 2023
Event30th Annual IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2023 - Goa, India
Duration: Dec 18 2023Dec 21 2023

Publication series

NameProceedings - 2023 IEEE 30th International Conference on High Performance Computing, Data, and Analytics, HiPC 2023

Conference

Conference30th Annual IEEE International Conference on High Performance Computing, Data, and Analytics, HiPC 2023
Country/TerritoryIndia
CityGoa
Period12/18/2312/21/23

Keywords

  • Data compression
  • High-performance computing
  • Machine learning

Fingerprint

Dive into the research topics of 'Fast Algorithms for Scientific Data Compression'. Together they form a unique fingerprint.

Cite this