An effective online data monitoring and saving strategy for large-scale climate simulations

Xiaochen Xian, Rick Archibald, Benjamin Mayer, Kaibo Liu, Jian Li

Research output: Contribution to journalArticlepeer-review

3 Scopus citations

Abstract

Large-scale climate simulation models have been developed and widely used to generate historical data and study future climate scenarios. These simulation models often have to run for a couple of months to understand the changes in the global climate over the course of decades. This long-duration simulation process creates a huge amount of data with both high temporal and spatial resolution information; however, how to effectively monitor and record the climate changes based on these large-scale simulation results that are continuously produced in real time still remains to be resolved. Due to the slow process of writing data to disk, the current practice is to save a snapshot of the simulation results at a constant, slow rate although the data generation process runs at a very high speed. This paper proposes an effective online data monitoring and saving strategy over the temporal and spatial domains with the consideration of practical storage and memory capacity constraints. Our proposed method is able to intelligently select and record the most informative extreme values in the raw data generated from real-time simulations in the context of better monitoring climate changes.

Original languageEnglish
Pages (from-to)330-346
Number of pages17
JournalQuality Technology and Quantitative Management
Volume16
Issue number3
DOIs
StatePublished - May 4 2019

Funding

This work was supported by the National Science Foundation [grant number NSF CMMI-1362529]; the Air Force Office of Scientific Research; National Natural Science Foundation of China [grant number 71402133], [grant number 71602155], [grant number 71572138], [grant number 11501209]. The submitted manuscript is based upon work, authored in part by contractors [UT-Battelle LLC, manager of Oak Ridge National Laboratory (ORNL)], and supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Applied Mathematics program. Accordingly, the U.S. Government retains a non-exclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes. This work was supported by the National Science Foundation [grant number NSF CMMI-1362529]; the Air Force Office of Scientific Research; National Natural Science Foundation of China [grant number 71402133], [grant number 71602155], [grant number 71572138], [grant number 11501209]. The submitted manuscript is based upon work, authored in part by contractors [UT-Battelle LLC, manager of Oak Ridge National Laboratory (ORNL)], and supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Applied Mathematics program. Accordingly, the U.S. Government retains a non-exclusive, royalty-free license to publish or reproduce the published form of this contribution, or allow others to do so, for U.S. Government purposes.

Keywords

  • Big data
  • local extrema
  • raw simulation data
  • spatial and temporal domains

Fingerprint

Dive into the research topics of 'An effective online data monitoring and saving strategy for large-scale climate simulations'. Together they form a unique fingerprint.

Cite this