Mediating Data Center Storage Diversity in HPC Applications with FAODEL

Patrick Widener; Craig Ulmer; Scott Levy; Todd Kordenbrock; Gary Templet

doi:10.1007/978-3-030-34356-9_22

Mediating Data Center Storage Diversity in HPC Applications with FAODEL

Patrick Widener, Craig Ulmer, Scott Levy, Todd Kordenbrock, Gary Templet

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Composition of computational science applications into both ad hoc pipelines for analysis of collected or generated data and into well-defined and repeatable workflows is becoming increasingly popular. Meanwhile, dedicated high performance computing storage environments are rapidly becoming more diverse, with both significant amounts of non-volatile memory storage and mature parallel file systems available. At the same time, computational science codes are being coupled to data analysis tools which are not filesystem-oriented. In this paper, we describe how the FAODEL data management service can expose different available data storage options and mediate among them in both application- and FAODEL-directed ways. These capabilities allow applications to exploit their knowledge of the different types of data they may exchange during a workflow execution, and also provide FAODEL with mechanisms to proactively tune data storage behavior when appropriate. We describe the implementation of these capabilities in FAODEL and how they are used by applications, and present preliminary performance results demonstrating the potential benefits of our approach.

Original language	English
Title of host publication	High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers
Editors	Michèle Weiland, Guido Juckeland, Sadaf Alam, Heike Jagode
Publisher	Springer
Pages	275-287
Number of pages	13
ISBN (Print)	9783030343552
DOIs	https://doi.org/10.1007/978-3-030-34356-9_22
State	Published - 2019
Externally published	Yes
Event	34th International Conference on High Performance Computing, ISC High Performance 2019 - Frankfurt, Germany Duration: Jun 16 2019 → Jun 20 2019

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11887 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	34th International Conference on High Performance Computing, ISC High Performance 2019
Country/Territory	Germany
City	Frankfurt
Period	06/16/19 → 06/20/19

Keywords

Composition
Data management
Scalability
Workflow

Access to Document

10.1007/978-3-030-34356-9_22

Cite this

Widener, P., Ulmer, C., Levy, S., Kordenbrock, T., & Templet, G. (2019). Mediating Data Center Storage Diversity in HPC Applications with FAODEL. In M. Weiland, G. Juckeland, S. Alam, & H. Jagode (Eds.), High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers (pp. 275-287). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11887 LNCS). Springer. https://doi.org/10.1007/978-3-030-34356-9_22

Widener, Patrick ; Ulmer, Craig ; Levy, Scott et al. / Mediating Data Center Storage Diversity in HPC Applications with FAODEL. High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers. editor / Michèle Weiland ; Guido Juckeland ; Sadaf Alam ; Heike Jagode. Springer, 2019. pp. 275-287 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{307b904088fd411489000cd43fc6e101,

title = "Mediating Data Center Storage Diversity in HPC Applications with FAODEL",

abstract = "Composition of computational science applications into both ad hoc pipelines for analysis of collected or generated data and into well-defined and repeatable workflows is becoming increasingly popular. Meanwhile, dedicated high performance computing storage environments are rapidly becoming more diverse, with both significant amounts of non-volatile memory storage and mature parallel file systems available. At the same time, computational science codes are being coupled to data analysis tools which are not filesystem-oriented. In this paper, we describe how the FAODEL data management service can expose different available data storage options and mediate among them in both application- and FAODEL-directed ways. These capabilities allow applications to exploit their knowledge of the different types of data they may exchange during a workflow execution, and also provide FAODEL with mechanisms to proactively tune data storage behavior when appropriate. We describe the implementation of these capabilities in FAODEL and how they are used by applications, and present preliminary performance results demonstrating the potential benefits of our approach.",

keywords = "Composition, Data management, Scalability, Workflow",

author = "Patrick Widener and Craig Ulmer and Scott Levy and Todd Kordenbrock and Gary Templet",

note = "Publisher Copyright: {\textcopyright} 2019, Springer Nature Switzerland AG.; 34th International Conference on High Performance Computing, ISC High Performance 2019 ; Conference date: 16-06-2019 Through 20-06-2019",

year = "2019",

doi = "10.1007/978-3-030-34356-9_22",

language = "English",

isbn = "9783030343552",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer",

pages = "275--287",

editor = "Mich{\`e}le Weiland and Guido Juckeland and Sadaf Alam and Heike Jagode",

booktitle = "High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers",

}

Widener, P, Ulmer, C, Levy, S, Kordenbrock, T & Templet, G 2019, Mediating Data Center Storage Diversity in HPC Applications with FAODEL. in M Weiland, G Juckeland, S Alam & H Jagode (eds), High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11887 LNCS, Springer, pp. 275-287, 34th International Conference on High Performance Computing, ISC High Performance 2019, Frankfurt, Germany, 06/16/19. https://doi.org/10.1007/978-3-030-34356-9_22

Mediating Data Center Storage Diversity in HPC Applications with FAODEL. / Widener, Patrick; Ulmer, Craig; Levy, Scott et al.
High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers. ed. / Michèle Weiland; Guido Juckeland; Sadaf Alam; Heike Jagode. Springer, 2019. p. 275-287 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11887 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Mediating Data Center Storage Diversity in HPC Applications with FAODEL

AU - Widener, Patrick

AU - Ulmer, Craig

AU - Levy, Scott

AU - Kordenbrock, Todd

AU - Templet, Gary

PY - 2019

Y1 - 2019

N2 - Composition of computational science applications into both ad hoc pipelines for analysis of collected or generated data and into well-defined and repeatable workflows is becoming increasingly popular. Meanwhile, dedicated high performance computing storage environments are rapidly becoming more diverse, with both significant amounts of non-volatile memory storage and mature parallel file systems available. At the same time, computational science codes are being coupled to data analysis tools which are not filesystem-oriented. In this paper, we describe how the FAODEL data management service can expose different available data storage options and mediate among them in both application- and FAODEL-directed ways. These capabilities allow applications to exploit their knowledge of the different types of data they may exchange during a workflow execution, and also provide FAODEL with mechanisms to proactively tune data storage behavior when appropriate. We describe the implementation of these capabilities in FAODEL and how they are used by applications, and present preliminary performance results demonstrating the potential benefits of our approach.

AB - Composition of computational science applications into both ad hoc pipelines for analysis of collected or generated data and into well-defined and repeatable workflows is becoming increasingly popular. Meanwhile, dedicated high performance computing storage environments are rapidly becoming more diverse, with both significant amounts of non-volatile memory storage and mature parallel file systems available. At the same time, computational science codes are being coupled to data analysis tools which are not filesystem-oriented. In this paper, we describe how the FAODEL data management service can expose different available data storage options and mediate among them in both application- and FAODEL-directed ways. These capabilities allow applications to exploit their knowledge of the different types of data they may exchange during a workflow execution, and also provide FAODEL with mechanisms to proactively tune data storage behavior when appropriate. We describe the implementation of these capabilities in FAODEL and how they are used by applications, and present preliminary performance results demonstrating the potential benefits of our approach.

KW - Composition

KW - Data management

KW - Scalability

KW - Workflow

UR - http://www.scopus.com/inward/record.url?scp=85076844717&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-34356-9_22

DO - 10.1007/978-3-030-34356-9_22

M3 - Conference contribution

AN - SCOPUS:85076844717

SN - 9783030343552

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 275

EP - 287

BT - High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers

A2 - Weiland, Michèle

A2 - Juckeland, Guido

A2 - Alam, Sadaf

A2 - Jagode, Heike

PB - Springer

T2 - 34th International Conference on High Performance Computing, ISC High Performance 2019

Y2 - 16 June 2019 through 20 June 2019

ER -

Widener P, Ulmer C, Levy S, Kordenbrock T, Templet G. Mediating Data Center Storage Diversity in HPC Applications with FAODEL. In Weiland M, Juckeland G, Alam S, Jagode H, editors, High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers. Springer. 2019. p. 275-287. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-34356-9_22

Mediating Data Center Storage Diversity in HPC Applications with FAODEL

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this