Persistent data staging services for data intensive in-situ scientific workflows

Melissa Romanus; Fan Zhang; Tong Jin; Qian Sun; Hoang Bui; Manish Parashar; Jong Choi; Saloman Janhunen; Robert Hager; Scott Klasky; Choong Seock Chang; Ivan Rodero

doi:10.1145/2912152.2912157

Persistent data staging services for data intensive in-situ scientific workflows

Melissa Romanus, Fan Zhang, Tong Jin, Qian Sun, Hoang Bui, Manish Parashar, Jong Choi, Saloman Janhunen, Robert Hager, Scott Klasky, Choong Seock Chang, Ivan Rodero

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and data-related overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements. In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.

Original language	English
Title of host publication	DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing
Publisher	Association for Computing Machinery, Inc
Pages	37-44
Number of pages	8
ISBN (Electronic)	9781450343527
DOIs	https://doi.org/10.1145/2912152.2912157
State	Published - Jun 1 2016
Event	6th ACM International Workshop on Data-Intensive Distributed Computing, DIDC 2016 - Kyoto, Japan Duration: Jun 1 2016 → …

Publication series

Name	DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing

Conference

Conference	6th ACM International Workshop on Data-Intensive Distributed Computing, DIDC 2016
Country/Territory	Japan
City	Kyoto
Period	06/1/16 → …

Funding

The research presented in this work is supported in part by National Science Foundation (NSF) via grant numbers CNS 1305375, ACI 1339036, ACI 1310283, ACI 1441376, ACI 1464317 and IIS 1546145, and by the Director, Office of Advanced Scientific Computing Research, Office of Science, of the US Department of Energy Scientific Discovery through Advanced Computing (SciDAC) Institute for Scalable Data Management, Analysis and Visualization (SDAV) under award number DE-SC0007455, the DoE RSVP grant via subcontract number 4000126989 from UT Battelle, the Advanced Scientific Computing Research and Fusion Energy Sciences Partnership for Edge Physics Simulations (EPSI) under award number DE-FG02-06ER54857, the ExaCT Combustion Co-Design Center via subcontract number 4000110839 from UT Battelle, and via grant number DE-FOA-0001338, Storage Systems and Input/Output for Extreme Scale Science. The research at Rutgers was conducted as part of the Rutgers Discovery Informatics Institute (RDI2).

Funders	Funder number
Advanced Scientific Computing Research and Fusion Energy Sciences Partnership for Edge Physics Simulations
DoE RSVP	4000126989
EPSI	DE-FG02-06ER54857
ExaCT Combustion Co-Design Center	4000110839, DE-FOA-0001338
US Department of Energy	DE-SC0007455
National Science Foundation	1546145, CNS 1305375, 1464317, ACI 1310283, ACI 1339036, IIS 1546145, 1339036, 1310283, ACI 1441376, ACI 1464317, 1305375
Battelle
Office of Science
Advanced Scientific Computing Research

Access to Document

10.1145/2912152.2912157

Cite this

Romanus, M., Zhang, F., Jin, T., Sun, Q., Bui, H., Parashar, M., Choi, J., Janhunen, S., Hager, R., Klasky, S., Chang, C. S., & Rodero, I. (2016). Persistent data staging services for data intensive in-situ scientific workflows. In DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing (pp. 37-44). (DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/2912152.2912157

Romanus, Melissa ; Zhang, Fan ; Jin, Tong et al. / Persistent data staging services for data intensive in-situ scientific workflows. DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing. Association for Computing Machinery, Inc, 2016. pp. 37-44 (DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing).

@inproceedings{5b4aef4b106446c0a463c43dd125e8d9,

title = "Persistent data staging services for data intensive in-situ scientific workflows",

abstract = "Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and data-related overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements. In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.",

author = "Melissa Romanus and Fan Zhang and Tong Jin and Qian Sun and Hoang Bui and Manish Parashar and Jong Choi and Saloman Janhunen and Robert Hager and Scott Klasky and Chang, {Choong Seock} and Ivan Rodero",

note = "Publisher Copyright: {\textcopyright} 2016 ACM.; 6th ACM International Workshop on Data-Intensive Distributed Computing, DIDC 2016 ; Conference date: 01-06-2016",

year = "2016",

month = jun,

day = "1",

doi = "10.1145/2912152.2912157",

language = "English",

series = "DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing",

publisher = "Association for Computing Machinery, Inc",

pages = "37--44",

booktitle = "DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing",

}

Romanus, M, Zhang, F, Jin, T, Sun, Q, Bui, H, Parashar, M, Choi, J, Janhunen, S, Hager, R, Klasky, S, Chang, CS & Rodero, I 2016, Persistent data staging services for data intensive in-situ scientific workflows. in DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing. DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing, Association for Computing Machinery, Inc, pp. 37-44, 6th ACM International Workshop on Data-Intensive Distributed Computing, DIDC 2016, Kyoto, Japan, 06/1/16. https://doi.org/10.1145/2912152.2912157

Persistent data staging services for data intensive in-situ scientific workflows. / Romanus, Melissa; Zhang, Fan; Jin, Tong et al.
DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing. Association for Computing Machinery, Inc, 2016. p. 37-44 (DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Persistent data staging services for data intensive in-situ scientific workflows

AU - Romanus, Melissa

AU - Zhang, Fan

AU - Jin, Tong

AU - Sun, Qian

AU - Bui, Hoang

AU - Parashar, Manish

AU - Choi, Jong

AU - Janhunen, Saloman

AU - Hager, Robert

AU - Klasky, Scott

AU - Chang, Choong Seock

AU - Rodero, Ivan

PY - 2016/6/1

Y1 - 2016/6/1

N2 - Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and data-related overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements. In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.

AB - Scientific simulation workflows executing on very large scale computing systems are essential modalities for scientific investigation. The increasing scales and resolution of these simulations provide new opportunities for accurately modeling complex natural and engineered phenomena. However, the increasing complexity necessitates managing, transporting, and processing unprecedented amounts of data, and as a result, researchers are increasingly exploring data-staging and in-situ workflows to reduce data movement and data-related overheads. However, as these workflows become more dynamic in their structures and behaviors, data staging and in-situ solutions must evolve to support new requirements. In this paper, we explore how the service-oriented concept can be applied to extreme-scale in-situ workflows. Specifically, we explore persistent data staging as a service and present the design and implementation of DataSpaces as a Service, a service-oriented data staging framework. We use a dynamically coupled fusion simulation workflow to illustrate the capabilities of this framework and evaluate its performance and scalability.

UR - http://www.scopus.com/inward/record.url?scp=84978896007&partnerID=8YFLogxK

U2 - 10.1145/2912152.2912157

DO - 10.1145/2912152.2912157

M3 - Conference contribution

AN - SCOPUS:84978896007

T3 - DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing

SP - 37

EP - 44

BT - DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing

PB - Association for Computing Machinery, Inc

T2 - 6th ACM International Workshop on Data-Intensive Distributed Computing, DIDC 2016

Y2 - 1 June 2016

ER -

Romanus M, Zhang F, Jin T, Sun Q, Bui H, Parashar M et al. Persistent data staging services for data intensive in-situ scientific workflows. In DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing. Association for Computing Machinery, Inc. 2016. p. 37-44. (DIDC 2016 - Proceedings of the ACM International Workshop on Data-Intensive Distributed Computing). doi: 10.1145/2912152.2912157

Persistent data staging services for data intensive in-situ scientific workflows

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this