Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

Jonathan Bader; Jim Belak; Matthew Bement; Matthew Berry; Robert Carson; Daniela Cassol; Stephen Chan; John Coleman; Kastan Day; Alejandro Duque; Kjiersten Fagnan; Jeff Froula; Shantenu Jha; Daniel S. Katz; Piotr Kica; Volodymyr Kindratenko; Edward Kirton; Ramani Kothadia; Daniel Laney; Fabian Lehmann; Ulf Leser; Sabina Licholai; MacIej Malawski; Mario Melara; Elais Player; Matt Rolchigo; Setareh Sarrafan; Seung Jin Sul; Abdullah Syed; Lauritz Thamsen; Mikhail Titov; Matteo Turilli; Silvina Caino-Lores; Anirban Mandal

doi:10.1145/3624062.3626283

Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

Jonathan Bader, Jim Belak, Matthew Bement, Matthew Berry, Robert Carson, Daniela Cassol, Stephen Chan, John Coleman, Kastan Day, Alejandro Duque, Kjiersten Fagnan, Jeff Froula, Shantenu Jha, Daniel S. Katz, Piotr Kica, Volodymyr Kindratenko, Edward Kirton, Ramani Kothadia, Daniel Laney, Fabian LehmannUlf Leser, Sabina Licholai, MacIej Malawski, Mario Melara, Elais Player, Matt Rolchigo, Setareh Sarrafan, Seung Jin Sul, Abdullah Syed, Lauritz Thamsen, Mikhail Titov, Matteo Turilli, Silvina Caino-Lores, Anirban Mandal

Computational Coupled Physics

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

2 Scopus citations

Abstract

The annual Workshop on Workflows in Support of Large-Scale Science (WORKS) is a premier venue for the scientific workflow community to present the latest advances in research and development on the many facets of scientific workflows throughout their life-cycle. The Lightning Talks at WORKS focus on describing a novel tool, scientific workflow, or concept, which are work-in-progress and address emerging technologies and frameworks to foster discussion in the community. This paper summarizes the lightning talks at the 2023 edition of WORKS, covering five topics: leveraging large language models to build and execute workflows; developing a common workflow scheduler interface; scaling uncertainty workflow applications on exascale computing systems; evaluating a transcriptomics workflow for cloud vs. HPC systems; and best practices in migrating legacy workflows to workflow management systems.

Original language	English
Title of host publication	Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Publisher	Association for Computing Machinery
Pages	2097-2108
Number of pages	12
ISBN (Electronic)	9798400707858
DOIs	https://doi.org/10.1145/3624062.3626283
State	Published - Nov 12 2023
Event	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 - Denver, United States Duration: Nov 12 2023 → Nov 17 2023

Publication series

Name	ACM International Conference Proceeding Series

Conference

Conference	2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023
Country/Territory	United States
City	Denver
Period	11/12/23 → 11/17/23

Funding

The materials this work is based upon are supported by NSF award 2050195; the German Research Foundation (DFG) CRC 1404: "FONDA: Foundations of Workflows for Large-Scale Scientific Data Analysis;" the Office of Science of the U.S. Department of Energy (DOE) under Contract No. DE-AC05-00OR22725; the Exascale Computing Project (ECP, 17-SC-20-SC) ExaWorks project under DOE Contract No. DE-SC0012704; Lawrence Livermore National Laboratory under Contract No. DE-AC52-07NA27344; DOE INCITE awards for allocations on Summit, Crusher, and Frontier; the European Union (EU) Horizon 2020 research and innovation programme under grant agreement Sano No. 857533 within the International Research Agendas programme of the Foundation for Polish Science, co-financed by the EU under the European Regional Development Fund; the EU Horizon 2020 research and innovation programme under grant agreement NEARDATA No. 101092644; the PLGrid Infrastructure and the Ares supercomputer at ACC Cyfronet AGH; and DOE Joint Genome Institute under Contract No. DE-AC02-05CH11231.

Keywords

Scientific workflows
cloud computing
high-performance computing
large language models
legacy workflows
workflow scheduling

Access to Document

10.1145/3624062.3626283

Cite this

Bader, J., Belak, J., Bement, M., Berry, M., Carson, R., Cassol, D., Chan, S., Coleman, J., Day, K., Duque, A., Fagnan, K., Froula, J., Jha, S., Katz, D. S., Kica, P., Kindratenko, V., Kirton, E., Kothadia, R., Laney, D., ... Mandal, A. (2023). Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 (pp. 2097-2108). (ACM International Conference Proceeding Series). Association for Computing Machinery. https://doi.org/10.1145/3624062.3626283

Bader, Jonathan ; Belak, Jim ; Bement, Matthew et al. / Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments. Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. pp. 2097-2108 (ACM International Conference Proceeding Series).

@inproceedings{6979b47dde8243a2819c98fe6595431b,

title = "Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments",

abstract = "The annual Workshop on Workflows in Support of Large-Scale Science (WORKS) is a premier venue for the scientific workflow community to present the latest advances in research and development on the many facets of scientific workflows throughout their life-cycle. The Lightning Talks at WORKS focus on describing a novel tool, scientific workflow, or concept, which are work-in-progress and address emerging technologies and frameworks to foster discussion in the community. This paper summarizes the lightning talks at the 2023 edition of WORKS, covering five topics: leveraging large language models to build and execute workflows; developing a common workflow scheduler interface; scaling uncertainty workflow applications on exascale computing systems; evaluating a transcriptomics workflow for cloud vs. HPC systems; and best practices in migrating legacy workflows to workflow management systems.",

keywords = "Scientific workflows, cloud computing, high-performance computing, large language models, legacy workflows, workflow scheduling",

author = "Jonathan Bader and Jim Belak and Matthew Bement and Matthew Berry and Robert Carson and Daniela Cassol and Stephen Chan and John Coleman and Kastan Day and Alejandro Duque and Kjiersten Fagnan and Jeff Froula and Shantenu Jha and Katz, \{Daniel S.\} and Piotr Kica and Volodymyr Kindratenko and Edward Kirton and Ramani Kothadia and Daniel Laney and Fabian Lehmann and Ulf Leser and Sabina Licholai and MacIej Malawski and Mario Melara and Elais Player and Matt Rolchigo and Setareh Sarrafan and Sul, \{Seung Jin\} and Abdullah Syed and Lauritz Thamsen and Mikhail Titov and Matteo Turilli and Silvina Caino-Lores and Anirban Mandal",

note = "Publisher Copyright: {\textcopyright} 2023 ACM.; 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023 ; Conference date: 12-11-2023 Through 17-11-2023",

year = "2023",

month = nov,

day = "12",

doi = "10.1145/3624062.3626283",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "2097--2108",

booktitle = "Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023",

}

Bader, J, Belak, J, Bement, M, Berry, M, Carson, R, Cassol, D, Chan, S, Coleman, J, Day, K, Duque, A, Fagnan, K, Froula, J, Jha, S, Katz, DS, Kica, P, Kindratenko, V, Kirton, E, Kothadia, R, Laney, D, Lehmann, F, Leser, U, Licholai, S, Malawski, M, Melara, M, Player, E, Rolchigo, M, Sarrafan, S, Sul, SJ, Syed, A, Thamsen, L, Titov, M, Turilli, M, Caino-Lores, S & Mandal, A 2023, Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments. in Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. ACM International Conference Proceeding Series, Association for Computing Machinery, pp. 2097-2108, 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023, Denver, United States, 11/12/23. https://doi.org/10.1145/3624062.3626283

Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments. / Bader, Jonathan; Belak, Jim; Bement, Matthew et al.
Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery, 2023. p. 2097-2108 (ACM International Conference Proceeding Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

AU - Bader, Jonathan

AU - Belak, Jim

AU - Bement, Matthew

AU - Berry, Matthew

AU - Carson, Robert

AU - Cassol, Daniela

AU - Chan, Stephen

AU - Coleman, John

AU - Day, Kastan

AU - Duque, Alejandro

AU - Fagnan, Kjiersten

AU - Froula, Jeff

AU - Jha, Shantenu

AU - Katz, Daniel S.

AU - Kica, Piotr

AU - Kindratenko, Volodymyr

AU - Kirton, Edward

AU - Kothadia, Ramani

AU - Laney, Daniel

AU - Lehmann, Fabian

AU - Leser, Ulf

AU - Licholai, Sabina

AU - Malawski, MacIej

AU - Melara, Mario

AU - Player, Elais

AU - Rolchigo, Matt

AU - Sarrafan, Setareh

AU - Sul, Seung Jin

AU - Syed, Abdullah

AU - Thamsen, Lauritz

AU - Titov, Mikhail

AU - Turilli, Matteo

AU - Caino-Lores, Silvina

AU - Mandal, Anirban

PY - 2023/11/12

Y1 - 2023/11/12

N2 - The annual Workshop on Workflows in Support of Large-Scale Science (WORKS) is a premier venue for the scientific workflow community to present the latest advances in research and development on the many facets of scientific workflows throughout their life-cycle. The Lightning Talks at WORKS focus on describing a novel tool, scientific workflow, or concept, which are work-in-progress and address emerging technologies and frameworks to foster discussion in the community. This paper summarizes the lightning talks at the 2023 edition of WORKS, covering five topics: leveraging large language models to build and execute workflows; developing a common workflow scheduler interface; scaling uncertainty workflow applications on exascale computing systems; evaluating a transcriptomics workflow for cloud vs. HPC systems; and best practices in migrating legacy workflows to workflow management systems.

AB - The annual Workshop on Workflows in Support of Large-Scale Science (WORKS) is a premier venue for the scientific workflow community to present the latest advances in research and development on the many facets of scientific workflows throughout their life-cycle. The Lightning Talks at WORKS focus on describing a novel tool, scientific workflow, or concept, which are work-in-progress and address emerging technologies and frameworks to foster discussion in the community. This paper summarizes the lightning talks at the 2023 edition of WORKS, covering five topics: leveraging large language models to build and execute workflows; developing a common workflow scheduler interface; scaling uncertainty workflow applications on exascale computing systems; evaluating a transcriptomics workflow for cloud vs. HPC systems; and best practices in migrating legacy workflows to workflow management systems.

KW - Scientific workflows

KW - cloud computing

KW - high-performance computing

KW - large language models

KW - legacy workflows

KW - workflow scheduling

UR - https://www.scopus.com/pages/publications/85178140885

U2 - 10.1145/3624062.3626283

DO - 10.1145/3624062.3626283

M3 - Conference contribution

AN - SCOPUS:85178140885

T3 - ACM International Conference Proceeding Series

SP - 2097

EP - 2108

BT - Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

PB - Association for Computing Machinery

T2 - 2023 International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023

Y2 - 12 November 2023 through 17 November 2023

ER -

Bader J, Belak J, Bement M, Berry M, Carson R, Cassol D et al. Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments. In Proceedings of 2023 SC Workshops of the International Conference on High Performance Computing, Network, Storage, and Analysis, SC Workshops 2023. Association for Computing Machinery. 2023. p. 2097-2108. (ACM International Conference Proceeding Series). doi: 10.1145/3624062.3626283

Novel Approaches Toward Scalable Composable Workflows in Hyper-Heterogeneous Computing Environments

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this