A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

Sterling Smith; Emily Belli; Orso Meneghini; Reuben Budiardja; David Schissel; Jeff Candy; Tom Neiser; Adam Eubanks

doi:10.1007/978-3-031-23606-8_6

A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

Sterling Smith, Emily Belli, Orso Meneghini, Reuben Budiardja, David Schissel, Jeff Candy, Tom Neiser, Adam Eubanks

Advanced Computing for Nuclear, Particle

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. The ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

Original language	English
Title of host publication	Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers
Editors	Kothe Doug, Geist Al, Swaroop Pophale, Hong Liu, Suzanne Parete-Koon
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	87-100
Number of pages	14
ISBN (Print)	9783031236051
DOIs	https://doi.org/10.1007/978-3-031-23606-8_6
State	Published - 2022
Event	Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022 - Virtual, Online Duration: Aug 24 2022 → Aug 25 2022

Publication series

Name	Communications in Computer and Information Science
Volume	1690 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022
City	Virtual, Online
Period	08/24/22 → 08/25/22

Funding

Acknowledgments. This work was supported by the U.S. Department of Energy under awards DE-FC02-04ER54698, DE-FG02-95ER54309, DE-FC02-06ER54873, and DE-SC0017992. Some material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Fusion Energy Sciences, using the DIII-D National Fusion Facility, a DOE Office of Science user facility, under Award(s) DE-FC02-04ER54698. Computing resources were also provided by the National Energy Research Scientific Computing Center, which is an Office of Science User Facility supported under Contract DE-AC02-05CH11231. An award of computer time was also provided by the INCITE program. This research used resources of the Oak Ridge Leadership Computing Facility, which is an Office of Science User Facility supported under Contract DE-AC05-00OR22725. This report was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor any agency thereof, nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial product, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or any agency thereof. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or any agency thereof. In 2007, a database of over 300 nonlinear simulations [14] based on the GYRO gyrokinetic code was successfully used to calibrate the original, widely-used TGLF reduced transport model [4]. The subsequent speed and accuracy of TGLF enabled a revolution in the accuracy of modeling calculations. The original GYRO simulations incorporated only ion-scale physics, typically assumed a pure plasma (deuterium ions + electrons), and were representative of weakly-shaped core plasma parameters. Several new gyrokinetic database efforts are underway, including the US CGYRODB database at NERSC based on the CGYRO code [12], the US MGKDB [15] at NERSC based on the GENE code [16], and the European Gyro-Kinetic Database Project (GKDB) hosted at gkdb.org based on the GKW [17] and GENE [16] codes. One goal of the gyrokinetic database projects is to store not only the physics input paramaters and selected output data, but also all code-specific model and resolution parameters for reproducibility. The two US databases, CGYRODB and MGKDB, were established and are currently directly organized through projects funded by the US DOE SciDAC (Scientific Discovery through Advanced Computing) program, whose aim is to bring together physicists, computer scientists, and mathematicians to develop new computational methods for solving challenging scientific problems. These projects are AToM: Advanced Tokamak Modeling Environment (for CGYRODB) and Partnership for Multiscale Gyrokinetic (MGK) Turbulence (for MGKDB). Both CGYRODB and MGKDB are presently hosted by Community File System (CFS) storage space at NERSC, where most of the simulations are also run with computational time in support of the associated SciDAC grants. Since the present database consists mostly of smaller-scale nonlinear simulations (e.g. ion-scale or electron-scale spatial resolution) with only highly-distilled output data, the CFS project space is presently sufficient. For example, the directory storage quota for the CGYRODB at NERSC is only 4 TB. A typical dataset for a single CGYRO ion-scale dataset is on the order of 0.1 GB.

Keywords

Big data
Data management
Fusion
High performance computing

Access to Document

10.1007/978-3-031-23606-8_6

Cite this

Smith, S., Belli, E., Meneghini, O., Budiardja, R., Schissel, D., Candy, J., Neiser, T., & Eubanks, A. (2022). A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management. In K. Doug, G. Al, S. Pophale, H. Liu, & S. Parete-Koon (Eds.), Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers (pp. 87-100). (Communications in Computer and Information Science; Vol. 1690 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-23606-8_6

Smith, Sterling ; Belli, Emily ; Meneghini, Orso et al. / A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management. Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers. editor / Kothe Doug ; Geist Al ; Swaroop Pophale ; Hong Liu ; Suzanne Parete-Koon. Springer Science and Business Media Deutschland GmbH, 2022. pp. 87-100 (Communications in Computer and Information Science).

@inproceedings{a244b41ff3b541b4a7cc59661125d49c,

title = "A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management",

abstract = "The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. The ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.",

keywords = "Big data, Data management, Fusion, High performance computing",

author = "Sterling Smith and Emily Belli and Orso Meneghini and Reuben Budiardja and David Schissel and Jeff Candy and Tom Neiser and Adam Eubanks",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.; Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022 ; Conference date: 24-08-2022 Through 25-08-2022",

year = "2022",

doi = "10.1007/978-3-031-23606-8_6",

language = "English",

isbn = "9783031236051",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "87--100",

editor = "Kothe Doug and Geist Al and Swaroop Pophale and Hong Liu and Suzanne Parete-Koon",

booktitle = "Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers",

}

Smith, S, Belli, E, Meneghini, O, Budiardja, R, Schissel, D, Candy, J, Neiser, T & Eubanks, A 2022, A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management. in K Doug, G Al, S Pophale, H Liu & S Parete-Koon (eds), Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers. Communications in Computer and Information Science, vol. 1690 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 87-100, Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Virtual, Online, 08/24/22. https://doi.org/10.1007/978-3-031-23606-8_6

A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management. / Smith, Sterling; Belli, Emily; Meneghini, Orso et al.
Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers. ed. / Kothe Doug; Geist Al; Swaroop Pophale; Hong Liu; Suzanne Parete-Koon. Springer Science and Business Media Deutschland GmbH, 2022. p. 87-100 (Communications in Computer and Information Science; Vol. 1690 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management

AU - Smith, Sterling

AU - Belli, Emily

AU - Meneghini, Orso

AU - Budiardja, Reuben

AU - Schissel, David

AU - Candy, Jeff

AU - Neiser, Tom

AU - Eubanks, Adam

PY - 2022

Y1 - 2022

N2 - The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. The ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

AB - The operation of large US Department of Energy (DOE) research facilities, like the DIII-D National Fusion Facility, results in the collection of complex multi-dimensional scientific datasets, both experimental and model-generated. In the future, it is envisioned that integrated data analysis coupled with large-scale high performance computing (HPC) simulations will be used to improve experimental planning and operation. Practically, massive data sets from these simulations provide the physics basis for generation of both reduced semi-analytic and machine-learning-based models. Storage of both HPC simulation datasets (generated from US DOE leadership computing facilities) and experimental datasets presents significant challenges. In this paper, we present a vision for a DOE-wide data management workflow that integrates US DOE fusion facilities with leadership computing facilities. Data persistence and long-term availability beyond the length of allocated projects is essential, particularly for verification and recalibration of artificial intelligence and machine learning (AI/ML) models. Because these data sets are often generated and shared among hundreds of users across multiple leadership computing facility centers, they would benefit from cross-platform accessibility, persistent identifiers (e.g. DOI, or digital object identifier), and provenance tracking. The ability to handle different data access patterns suggests that a combination of low cost, high latency (e.g. for storing ML training sets) and high cost, low latency systems (e.g. for real-time, integrated machine control feedback) may be needed.

KW - Big data

KW - Data management

KW - Fusion

KW - High performance computing

UR - http://www.scopus.com/inward/record.url?scp=85148688912&partnerID=8YFLogxK

U2 - 10.1007/978-3-031-23606-8_6

DO - 10.1007/978-3-031-23606-8_6

M3 - Conference contribution

AN - SCOPUS:85148688912

SN - 9783031236051

T3 - Communications in Computer and Information Science

SP - 87

EP - 100

BT - Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers

A2 - Doug, Kothe

A2 - Al, Geist

A2 - Pophale, Swaroop

A2 - Liu, Hong

A2 - Parete-Koon, Suzanne

PB - Springer Science and Business Media Deutschland GmbH

T2 - Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022

Y2 - 24 August 2022 through 25 August 2022

ER -

Smith S, Belli E, Meneghini O, Budiardja R, Schissel D, Candy J et al. A Vision for Coupling Operation of US Fusion Facilities with HPC Systems and the Implications for Workflows and Data Management. In Doug K, Al G, Pophale S, Liu H, Parete-Koon S, editors, Accelerating Science and Engineering Discoveries Through Integrated Research Infrastructure for Experiment, Big Data, Modeling and Simulation - 22nd Smoky Mountains Computational Sciences and Engineering Conference, SMC 2022, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2022. p. 87-100. (Communications in Computer and Information Science). doi: 10.1007/978-3-031-23606-8_6