A Community Roadmap for Scientific Workflows Research and Development

Rafael Ferreira Da Silva; Henri Casanova; Kyle Chard; Ilkay Altintas; Rosa M. Badia; Bartosz Balis; Taina Coleman; Frederik Coppens; Frank Di Natale; Bjoern Enders; Thomas Fahringer; Rosa Filgueira; Grigori Fursin; Daniel Garijo; Carole Goble; Dorran Howell; Shantenu Jha; Daniel S. Katz; Daniel Laney; Ulf Leser; MacIej Malawski; Kshitij Mehta; Loic Pottier; Jonathan Ozik; J. Luc Peterson; Lavanya Ramakrishnan; Stian Soiland-Reyes; Douglas Thain; Matthew Wolf

doi:10.1109/WORKS54523.2021.00016

A Community Roadmap for Scientific Workflows Research and Development

Rafael Ferreira Da Silva, Henri Casanova, Kyle Chard, Ilkay Altintas, Rosa M. Badia, Bartosz Balis, Taina Coleman, Frederik Coppens, Frank Di Natale, Bjoern Enders, Thomas Fahringer, Rosa Filgueira, Grigori Fursin, Daniel Garijo, Carole Goble, Dorran Howell, Shantenu Jha, Daniel S. Katz, Daniel Laney, Ulf LeserMacIej Malawski, Kshitij Mehta, Loic Pottier, Jonathan Ozik, J. Luc Peterson, Lavanya Ramakrishnan, Stian Soiland-Reyes, Douglas Thain, Matthew Wolf

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

22 Scopus citations

Abstract

The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual 'Workflows Community Summits' (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

Original language	English
Title of host publication	Proceedings of WORKS 2021
Subtitle of host publication	16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	81-90
Number of pages	10
ISBN (Electronic)	9781665411363
DOIs	https://doi.org/10.1109/WORKS54523.2021.00016
State	Published - 2021
Event	16th IEEE Workshop on Workflows in Support of Large-Scale Science, WORKS 2021 - St. Louis, United States Duration: Nov 15 2021 → …

Publication series

Name	Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	16th IEEE Workshop on Workflows in Support of Large-Scale Science, WORKS 2021
Country/Territory	United States
City	St. Louis
Period	11/15/21 → …

Funding

This work was funded by NSF awards #2016610, #2016682, and #2016619, and by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy (DOE) Office of Science and the National Nuclear Security Administration (NNSA). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. DOE under Contract #DE-AC05-00OR22725.

Funders	Funder number
National Science Foundation	17-SC-20-SC, 2016682, 2016619, 2016610
U.S. Department of Energy
Office of Science
National Nuclear Security Administration	-AC05-00OR22725

Keywords

AI workflows
Community roadmap
Data management
Exascale computing
Interoperability
Scientific workflows

Access to Document

10.1109/WORKS54523.2021.00016

Cite this

Da Silva, R. F., Casanova, H., Chard, K., Altintas, I., Badia, R. M., Balis, B., Coleman, T., Coppens, F., Di Natale, F., Enders, B., Fahringer, T., Filgueira, R., Fursin, G., Garijo, D., Goble, C., Howell, D., Jha, S., Katz, D. S., Laney, D., ... Wolf, M. (2021). A Community Roadmap for Scientific Workflows Research and Development. In Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 81-90). (Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/WORKS54523.2021.00016

Da Silva, Rafael Ferreira ; Casanova, Henri ; Chard, Kyle et al. / A Community Roadmap for Scientific Workflows Research and Development. Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 81-90 (Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{264912ce9ab2409b947ad44a16dbf26c,

title = "A Community Roadmap for Scientific Workflows Research and Development",

abstract = "The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual 'Workflows Community Summits' (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.",

keywords = "AI workflows, Community roadmap, Data management, Exascale computing, Interoperability, Scientific workflows",

author = "{Da Silva}, {Rafael Ferreira} and Henri Casanova and Kyle Chard and Ilkay Altintas and Badia, {Rosa M.} and Bartosz Balis and Taina Coleman and Frederik Coppens and {Di Natale}, Frank and Bjoern Enders and Thomas Fahringer and Rosa Filgueira and Grigori Fursin and Daniel Garijo and Carole Goble and Dorran Howell and Shantenu Jha and Katz, {Daniel S.} and Daniel Laney and Ulf Leser and MacIej Malawski and Kshitij Mehta and Loic Pottier and Jonathan Ozik and {Luc Peterson}, J. and Lavanya Ramakrishnan and Stian Soiland-Reyes and Douglas Thain and Matthew Wolf",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 16th IEEE Workshop on Workflows in Support of Large-Scale Science, WORKS 2021 ; Conference date: 15-11-2021",

year = "2021",

doi = "10.1109/WORKS54523.2021.00016",

language = "English",

series = "Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "81--90",

booktitle = "Proceedings of WORKS 2021",

}

Da Silva, RF, Casanova, H, Chard, K, Altintas, I, Badia, RM, Balis, B, Coleman, T, Coppens, F, Di Natale, F, Enders, B, Fahringer, T, Filgueira, R, Fursin, G, Garijo, D, Goble, C, Howell, D, Jha, S, Katz, DS, Laney, D, Leser, U, Malawski, M, Mehta, K, Pottier, L, Ozik, J, Luc Peterson, J, Ramakrishnan, L, Soiland-Reyes, S, Thain, D & Wolf, M 2021, A Community Roadmap for Scientific Workflows Research and Development. in Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis. Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 81-90, 16th IEEE Workshop on Workflows in Support of Large-Scale Science, WORKS 2021, St. Louis, United States, 11/15/21. https://doi.org/10.1109/WORKS54523.2021.00016

A Community Roadmap for Scientific Workflows Research and Development. / Da Silva, Rafael Ferreira; Casanova, Henri; Chard, Kyle et al.
Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2021. p. 81-90 (Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Community Roadmap for Scientific Workflows Research and Development

AU - Da Silva, Rafael Ferreira

AU - Casanova, Henri

AU - Chard, Kyle

AU - Altintas, Ilkay

AU - Badia, Rosa M.

AU - Balis, Bartosz

AU - Coleman, Taina

AU - Coppens, Frederik

AU - Di Natale, Frank

AU - Enders, Bjoern

AU - Fahringer, Thomas

AU - Filgueira, Rosa

AU - Fursin, Grigori

AU - Garijo, Daniel

AU - Goble, Carole

AU - Howell, Dorran

AU - Jha, Shantenu

AU - Katz, Daniel S.

AU - Laney, Daniel

AU - Leser, Ulf

AU - Malawski, MacIej

AU - Mehta, Kshitij

AU - Pottier, Loic

AU - Ozik, Jonathan

AU - Luc Peterson, J.

AU - Ramakrishnan, Lavanya

AU - Soiland-Reyes, Stian

AU - Thain, Douglas

AU - Wolf, Matthew

PY - 2021

Y1 - 2021

N2 - The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual 'Workflows Community Summits' (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

AB - The landscape of workflow systems for scientific applications is notoriously convoluted with hundreds of seemingly equivalent workflow systems, many isolated research claims, and a steep learning curve. To address some of these challenges and lay the groundwork for transforming workflows research and development, the WorkflowsRI and ExaWorks projects partnered to bring the international workflows community together. This paper reports on discussions and findings from two virtual 'Workflows Community Summits' (January and April, 2021). The overarching goals of these workshops were to develop a view of the state of the art, identify crucial research challenges in the workflows community, articulate a vision for potential community efforts, and discuss technical approaches for realizing this vision. To this end, participants identified six broad themes: FAIR computational workflows; AI workflows; exascale challenges; APIs, interoperability, reuse, and standards; training and education; and building a workflows community. We summarize discussions and recommendations for each of these themes.

KW - AI workflows

KW - Community roadmap

KW - Data management

KW - Exascale computing

KW - Interoperability

KW - Scientific workflows

UR - http://www.scopus.com/inward/record.url?scp=85123475930&partnerID=8YFLogxK

U2 - 10.1109/WORKS54523.2021.00016

DO - 10.1109/WORKS54523.2021.00016

M3 - Conference contribution

AN - SCOPUS:85123475930

T3 - Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 81

EP - 90

BT - Proceedings of WORKS 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 16th IEEE Workshop on Workflows in Support of Large-Scale Science, WORKS 2021

Y2 - 15 November 2021

ER -

Da Silva RF, Casanova H, Chard K, Altintas I, Badia RM, Balis B et al. A Community Roadmap for Scientific Workflows Research and Development. In Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2021. p. 81-90. (Proceedings of WORKS 2021: 16th Workshop on Workflows in Support of Large-Scale Science, Held in conjunction with SC 2021: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/WORKS54523.2021.00016

A Community Roadmap for Scientific Workflows Research and Development

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this