Faodel: Data management for next-generation application workflows

Craig Ulmer; Shyamali Mukherjee; Gary Templet; Todd Kordenbrock; Scott Levy; Jay Lofstead; Patrick Widener; Margaret Lawson

doi:10.1145/3217880.3217888

Faodel: Data management for next-generation application workflows

Craig Ulmer, Shyamali Mukherjee, Gary Templet, Todd Kordenbrock, Scott Levy, Jay Lofstead, Patrick Widener, Margaret Lawson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

Composition of computational science applications, whether into ad hoc pipelines for analysis of simulation data or into well-defined and repeatable workflows, is becoming commonplace. In order to scale well as projected system and data sizes increase, developers will have to address a number of looming challenges. Increased contention for parallel filesystem bandwidth, accomodating in situ and ex situ processing, and the advent of decentralized programming models will all complicate application composition for next-generation systems. In this paper, we introduce a set of data services, Faodel, which provide scalable data management for workflows and composed applications. Faodel allows workflow components to directly and efficiently exchange data in semantically appropriate forms, rather than those dictated by the storage hierarchy or programming model in use. We describe the architecture of Faodel and present preliminary performance results demonstrating its potential for scalability in workflow scenarios.

Original language	English
Title of host publication	Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018
Publisher	Association for Computing Machinery, Inc
ISBN (Electronic)	9781450358637
DOIs	https://doi.org/10.1145/3217880.3217888
State	Published - Jun 11 2018
Externally published	Yes
Event	9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Tempe, United States Duration: Jun 11 2018 → …

Publication series

Name	Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018

Conference

Conference	9th Workshop on Scientific Cloud Computing, ScienceCloud 2018
Country/Territory	United States
City	Tempe
Period	06/11/18 → …

Funding

Sandia National Laboratories is a multimission laboratory managed and operated by National Technology and Engineering Solutions of Sandia, LLC, a wholly owned subsidiary of Honeywell International, Inc., for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-NA0003525.

Keywords

Composition
Data management
Scalability
Workflow

Access to Document

10.1145/3217880.3217888

Cite this

Ulmer, C., Mukherjee, S., Templet, G., Kordenbrock, T., Levy, S., Lofstead, J., Widener, P., & Lawson, M. (2018). Faodel: Data management for next-generation application workflows. In Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018 Article a8 (Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018). Association for Computing Machinery, Inc. https://doi.org/10.1145/3217880.3217888

Ulmer, Craig ; Mukherjee, Shyamali ; Templet, Gary et al. / Faodel : Data management for next-generation application workflows. Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc, 2018. (Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018).

@inproceedings{3fd01d5d2f5d4ff595472a18130b9ab5,

title = "Faodel: Data management for next-generation application workflows",

abstract = "Composition of computational science applications, whether into ad hoc pipelines for analysis of simulation data or into well-defined and repeatable workflows, is becoming commonplace. In order to scale well as projected system and data sizes increase, developers will have to address a number of looming challenges. Increased contention for parallel filesystem bandwidth, accomodating in situ and ex situ processing, and the advent of decentralized programming models will all complicate application composition for next-generation systems. In this paper, we introduce a set of data services, Faodel, which provide scalable data management for workflows and composed applications. Faodel allows workflow components to directly and efficiently exchange data in semantically appropriate forms, rather than those dictated by the storage hierarchy or programming model in use. We describe the architecture of Faodel and present preliminary performance results demonstrating its potential for scalability in workflow scenarios.",

keywords = "Composition, Data management, Scalability, Workflow",

author = "Craig Ulmer and Shyamali Mukherjee and Gary Templet and Todd Kordenbrock and Scott Levy and Jay Lofstead and Patrick Widener and Margaret Lawson",

note = "Publisher Copyright: {\textcopyright} 2018 Association for Computing Machinery.; 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 ; Conference date: 11-06-2018",

year = "2018",

month = jun,

day = "11",

doi = "10.1145/3217880.3217888",

language = "English",

series = "Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018",

publisher = "Association for Computing Machinery, Inc",

booktitle = "Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018",

}

Ulmer, C, Mukherjee, S, Templet, G, Kordenbrock, T, Levy, S, Lofstead, J, Widener, P & Lawson, M 2018, Faodel: Data management for next-generation application workflows. in Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018., a8, Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018, Association for Computing Machinery, Inc, 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018, Tempe, United States, 06/11/18. https://doi.org/10.1145/3217880.3217888

Faodel: Data management for next-generation application workflows. / Ulmer, Craig; Mukherjee, Shyamali; Templet, Gary et al.
Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc, 2018. a8 (Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Faodel

T2 - 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018

AU - Ulmer, Craig

AU - Mukherjee, Shyamali

AU - Templet, Gary

AU - Kordenbrock, Todd

AU - Levy, Scott

AU - Lofstead, Jay

AU - Widener, Patrick

AU - Lawson, Margaret

PY - 2018/6/11

Y1 - 2018/6/11

N2 - Composition of computational science applications, whether into ad hoc pipelines for analysis of simulation data or into well-defined and repeatable workflows, is becoming commonplace. In order to scale well as projected system and data sizes increase, developers will have to address a number of looming challenges. Increased contention for parallel filesystem bandwidth, accomodating in situ and ex situ processing, and the advent of decentralized programming models will all complicate application composition for next-generation systems. In this paper, we introduce a set of data services, Faodel, which provide scalable data management for workflows and composed applications. Faodel allows workflow components to directly and efficiently exchange data in semantically appropriate forms, rather than those dictated by the storage hierarchy or programming model in use. We describe the architecture of Faodel and present preliminary performance results demonstrating its potential for scalability in workflow scenarios.

AB - Composition of computational science applications, whether into ad hoc pipelines for analysis of simulation data or into well-defined and repeatable workflows, is becoming commonplace. In order to scale well as projected system and data sizes increase, developers will have to address a number of looming challenges. Increased contention for parallel filesystem bandwidth, accomodating in situ and ex situ processing, and the advent of decentralized programming models will all complicate application composition for next-generation systems. In this paper, we introduce a set of data services, Faodel, which provide scalable data management for workflows and composed applications. Faodel allows workflow components to directly and efficiently exchange data in semantically appropriate forms, rather than those dictated by the storage hierarchy or programming model in use. We describe the architecture of Faodel and present preliminary performance results demonstrating its potential for scalability in workflow scenarios.

KW - Composition

KW - Data management

KW - Scalability

KW - Workflow

UR - http://www.scopus.com/inward/record.url?scp=85050128756&partnerID=8YFLogxK

U2 - 10.1145/3217880.3217888

DO - 10.1145/3217880.3217888

M3 - Conference contribution

AN - SCOPUS:85050128756

T3 - Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018

BT - Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018

PB - Association for Computing Machinery, Inc

Y2 - 11 June 2018

ER -

Ulmer C, Mukherjee S, Templet G, Kordenbrock T, Levy S, Lofstead J et al. Faodel: Data management for next-generation application workflows. In Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018. Association for Computing Machinery, Inc. 2018. a8. (Proceedings of the 9th Workshop on Scientific Cloud Computing, ScienceCloud 2018 - Co-located with HPDC 2018). doi: 10.1145/3217880.3217888

Faodel: Data management for next-generation application workflows

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this