Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows

Tong Jin; Fan Zhang; Qian Sun; Hoang Bui; Melissa Romanus; Norbert Podhorszki; Scott Klasky; Hemanth Kolla; Jacqueline Chen; Robert Hager; Choong Seock Chang; Manish Parashar

doi:10.1109/IPDPS.2015.50

Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows

Tong Jin, Fan Zhang, Qian Sun, Hoang Bui, Melissa Romanus, Norbert Podhorszki, Scott Klasky, Hemanth Kolla, Jacqueline Chen, Robert Hager, Choong Seock Chang, Manish Parashar

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

35 Scopus citations

Abstract

As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.

Original language	English
Title of host publication	Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1033-1042
Number of pages	10
ISBN (Electronic)	9781479986484
DOIs	https://doi.org/10.1109/IPDPS.2015.50
State	Published - Jul 17 2015
Event	29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015 - Hyderabad, India Duration: May 25 2015 → May 29 2015

Publication series

Name	Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015

Conference

Conference	29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015
Country/Territory	India
City	Hyderabad
Period	05/25/15 → 05/29/15

Funding

The research presented in this work is supported in part by the US National Science Foundation (NSF) via grants numbers ACI 1339036, ACI 1310283, DMS 1228203 and IIP 0758566, by the Director, Office of Advanced Scientific Computing Research, Office of Science, of the U.S. Department of Energy through the Scientific Discovery through Advanced Computing (SciDAC) Institute of Scalable Data Management, Analysis and Visualization (SDAV) under award number DE-SC0007455, the Advanced Scientific Computing Research and Fusion Energy Sciences Partnership for Edge Physics Simulations (EPSI) under award number DE-FG02-06ER54857, the ExaCT Combustion Co-Design Center via subcontract number 4000110839 from UT Battelle, and the RSVP grant via subcontract number 4000126989 from UT Battelle. The research was conducted as part of the NSF Cloud and Autonomic Computing (CAC) Center at Rutgers University and the Rutgers Discovery Informatics Institute (RDI2).

Keywords

Coupling
Data management
adaptation
data placement
data staging
deep memory hierarchy
exascale
multi-tiered
ssd

Access to Document

10.1109/IPDPS.2015.50

Cite this

Jin, T., Zhang, F., Sun, Q., Bui, H., Romanus, M., Podhorszki, N., Klasky, S., Kolla, H., Chen, J., Hager, R., Chang, C. S., & Parashar, M. (2015). Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows. In Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015 (pp. 1033-1042). Article 7161588 (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IPDPS.2015.50

Jin, Tong ; Zhang, Fan ; Sun, Qian et al. / Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows. Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015. Institute of Electrical and Electronics Engineers Inc., 2015. pp. 1033-1042 (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015).

@inproceedings{976bec4d3c8d41c0a4dbed6e6adfa6fb,

title = "Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows",

abstract = "As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.",

keywords = "Coupling, Data management, adaptation, data placement, data staging, deep memory hierarchy, exascale, multi-tiered, ssd",

author = "Tong Jin and Fan Zhang and Qian Sun and Hoang Bui and Melissa Romanus and Norbert Podhorszki and Scott Klasky and Hemanth Kolla and Jacqueline Chen and Robert Hager and Chang, {Choong Seock} and Manish Parashar",

note = "Publisher Copyright: {\textcopyright} 2015 IEEE.; 29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015 ; Conference date: 25-05-2015 Through 29-05-2015",

year = "2015",

month = jul,

day = "17",

doi = "10.1109/IPDPS.2015.50",

language = "English",

series = "Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1033--1042",

booktitle = "Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015",

}

Jin, T, Zhang, F, Sun, Q, Bui, H, Romanus, M, Podhorszki, N , Klasky, S, Kolla, H, Chen, J, Hager, R, Chang, CS & Parashar, M 2015, Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows. in Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015., 7161588, Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015, Institute of Electrical and Electronics Engineers Inc., pp. 1033-1042, 29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015, Hyderabad, India, 05/25/15. https://doi.org/10.1109/IPDPS.2015.50

Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows. / Jin, Tong; Zhang, Fan; Sun, Qian et al.
Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015. Institute of Electrical and Electronics Engineers Inc., 2015. p. 1033-1042 7161588 (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows

AU - Jin, Tong

AU - Zhang, Fan

AU - Sun, Qian

AU - Bui, Hoang

AU - Romanus, Melissa

AU - Podhorszki, Norbert

AU - Klasky, Scott

AU - Kolla, Hemanth

AU - Chen, Jacqueline

AU - Hager, Robert

AU - Chang, Choong Seock

AU - Parashar, Manish

PY - 2015/7/17

Y1 - 2015/7/17

N2 - As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.

AB - As applications target extreme scales, data staging and in-situ/in-transit data processing have been proposed to address the data challenges and improve scientific discovery. However, further research is necessary in order to understand how growing data sizes from data intensive simulations coupled with the limited DRAM capacity in High End Computing systems will impact the effectiveness of this approach. In this paper, we explore how we can use deep memory levels for data staging, and develop a multi-tiered data staging method that spans bothDRAM and solid state disks (SSD). This approach allows us to support both code coupling and data management for data intensive simulation workflows. We also show how an adaptive application-aware data placement mechanism can dynamically manage and optimize data placement across the DRAM ands storage levels in this multi-tiered data staging method. We present an experimental evaluation of our approach using wolf resources: an Infiniband cluster (Sith) and a Cray XK7system (Titan), and using combustion (S3D) and fusion (XGC1) simulations.

KW - Coupling

KW - Data management

KW - adaptation

KW - data placement

KW - data staging

KW - deep memory hierarchy

KW - exascale

KW - multi-tiered

KW - ssd

UR - http://www.scopus.com/inward/record.url?scp=84966520332&partnerID=8YFLogxK

U2 - 10.1109/IPDPS.2015.50

DO - 10.1109/IPDPS.2015.50

M3 - Conference contribution

AN - SCOPUS:84966520332

T3 - Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015

SP - 1033

EP - 1042

BT - Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 29th IEEE International Parallel and Distributed Processing Symposium, IPDPS 2015

Y2 - 25 May 2015 through 29 May 2015

ER -

Jin T, Zhang F, Sun Q, Bui H, Romanus M, Podhorszki N et al. Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows. In Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015. Institute of Electrical and Electronics Engineers Inc. 2015. p. 1033-1042. 7161588. (Proceedings - 2015 IEEE 29th International Parallel and Distributed Processing Symposium, IPDPS 2015). doi: 10.1109/IPDPS.2015.50

Exploring Data Staging Across Deep Memory Hierarchies for Coupled Data Intensive Simulation Workflows

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this