Abstract
In the compression of scientific data, error-controlled compressors enable to considerably decrease the size of the dataset while maintaining adequate levels of accuracy. In this paper, we note that multi-level refactoring scheme such as MGARD i) rely on an approximation of the data based on the interpolation of coefficients, ii) estimate the resulting error with global metrics on the dataset. To improve on these two aspects, we propose a method that aims to divide the original dataset into blocks based on their smoothness and refactors each block separately with the most relevant interpolation order. We show the relevance of such a method on tailored datasets and the benefits and challenges when applying it to large scientific data.
| Original language | English |
|---|---|
| Title of host publication | Proceedings - 2024 IEEE International Conference on Big Data, BigData 2024 |
| Editors | Wei Ding, Chang-Tien Lu, Fusheng Wang, Liping Di, Kesheng Wu, Jun Huan, Raghu Nambiar, Jundong Li, Filip Ilievski, Ricardo Baeza-Yates, Xiaohua Hu |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| Pages | 4257-4264 |
| Number of pages | 8 |
| ISBN (Electronic) | 9798350362480 |
| DOIs | |
| State | Published - 2024 |
| Event | 2024 IEEE International Conference on Big Data, BigData 2024 - Washington, United States Duration: Dec 15 2024 → Dec 18 2024 |
Publication series
| Name | Proceedings - 2024 IEEE International Conference on Big Data, BigData 2024 |
|---|
Conference
| Conference | 2024 IEEE International Conference on Big Data, BigData 2024 |
|---|---|
| Country/Territory | United States |
| City | Washington |
| Period | 12/15/24 → 12/18/24 |
Funding
This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, worldwide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).