WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems

Henri Casanova; Kyle Berney; Serge Chastel; Rafael Ferreira Da Silva

doi:10.1109/COMPSAC57700.2023.00290

WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems

Henri Casanova, Kyle Berney, Serge Chastel, Rafael Ferreira Da Silva

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Scientific workflows have become ubiquitous across scientific fields, and their execution methods and systems continue to be the subject of research and development. Most experimental evaluations of these workflows rely on workflow instances, which can be either real-world or synthetic, to ensure relevance to current application domains or explore hypothetical/future scenarios. The WfCommons project addresses this need by providing data and tools to support such evaluations. In this paper, we present an overview of WfCommons and describe two recent developments. Firstly, we introduce a workflow execution "tracer"for Nextflow, which significantly enhances the set of real-world instances available in WfCommons. Secondly, we describe a workflow instance "translator"that enables the execution of any real-world or synthetic WfCommons workflow instance using Dask. Our contributions aim to provide researchers and practitioners with more comprehensive resources for evaluating scientific workflows.

Original language	English
Title of host publication	Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023
Editors	Hossain Shahriar, Yuuichi Teranishi, Alfredo Cuzzocrea, Moushumi Sharmin, Dave Towey, AKM Jahangir Alam Majumder, Hiroki Kashiwazaki, Ji-Jiang Yang, Michiharu Takemoto, Nazmus Sakib, Ryohei Banno, Sheikh Iqbal Ahamed
Publisher	IEEE Computer Society
Pages	1870-1875
Number of pages	6
ISBN (Electronic)	9798350326970
DOIs	https://doi.org/10.1109/COMPSAC57700.2023.00290
State	Published - 2023
Event	47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023 - Hybrid, Torino, Italy Duration: Jun 26 2023 → Jun 30 2023

Publication series

Name	Proceedings - International Computer Software and Applications Conference
Volume	2023-June
ISSN (Print)	0730-3157

Conference

Conference	47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023
Country/Territory	Italy
City	Hybrid, Torino
Period	06/26/23 → 06/30/23

Funding

This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, world-wide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). ACKNOWLEDGMENT This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This research was partially supported by NSF Awards #2106059 and #2103489. Finally, we thank the NSF Chameleon Cloud for providing time grants to access their resources.

Keywords

Scientific workflows
workflow instance collection
workflow instance execution

Access to Document

10.1109/COMPSAC57700.2023.00290

Cite this

Casanova, H., Berney, K., Chastel, S., & Da Silva, R. F. (2023). WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems. In H. Shahriar, Y. Teranishi, A. Cuzzocrea, M. Sharmin, D. Towey, AKM. J. A. Majumder, H. Kashiwazaki, J.-J. Yang, M. Takemoto, N. Sakib, R. Banno, & S. I. Ahamed (Eds.), Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023 (pp. 1870-1875). (Proceedings - International Computer Software and Applications Conference; Vol. 2023-June). IEEE Computer Society. https://doi.org/10.1109/COMPSAC57700.2023.00290

Casanova, Henri ; Berney, Kyle ; Chastel, Serge et al. / WfCommons : Data Collection and Runtime Experiments using Multiple Workflow Systems. Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. editor / Hossain Shahriar ; Yuuichi Teranishi ; Alfredo Cuzzocrea ; Moushumi Sharmin ; Dave Towey ; AKM Jahangir Alam Majumder ; Hiroki Kashiwazaki ; Ji-Jiang Yang ; Michiharu Takemoto ; Nazmus Sakib ; Ryohei Banno ; Sheikh Iqbal Ahamed. IEEE Computer Society, 2023. pp. 1870-1875 (Proceedings - International Computer Software and Applications Conference).

@inproceedings{d9e8b64a33c644b2aa1f33a2ddacd25f,

title = "WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems",

abstract = "Scientific workflows have become ubiquitous across scientific fields, and their execution methods and systems continue to be the subject of research and development. Most experimental evaluations of these workflows rely on workflow instances, which can be either real-world or synthetic, to ensure relevance to current application domains or explore hypothetical/future scenarios. The WfCommons project addresses this need by providing data and tools to support such evaluations. In this paper, we present an overview of WfCommons and describe two recent developments. Firstly, we introduce a workflow execution {"}tracer{"}for Nextflow, which significantly enhances the set of real-world instances available in WfCommons. Secondly, we describe a workflow instance {"}translator{"}that enables the execution of any real-world or synthetic WfCommons workflow instance using Dask. Our contributions aim to provide researchers and practitioners with more comprehensive resources for evaluating scientific workflows.",

keywords = "Scientific workflows, workflow instance collection, workflow instance execution",

author = "Henri Casanova and Kyle Berney and Serge Chastel and \{Da Silva\}, \{Rafael Ferreira\}",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023 ; Conference date: 26-06-2023 Through 30-06-2023",

year = "2023",

doi = "10.1109/COMPSAC57700.2023.00290",

language = "English",

series = "Proceedings - International Computer Software and Applications Conference",

publisher = "IEEE Computer Society",

pages = "1870--1875",

editor = "Hossain Shahriar and Yuuichi Teranishi and Alfredo Cuzzocrea and Moushumi Sharmin and Dave Towey and Majumder, \{AKM Jahangir Alam\} and Hiroki Kashiwazaki and Ji-Jiang Yang and Michiharu Takemoto and Nazmus Sakib and Ryohei Banno and Ahamed, \{Sheikh Iqbal\}",

booktitle = "Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023",

}

Casanova, H, Berney, K, Chastel, S & Da Silva, RF 2023, WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems. in H Shahriar, Y Teranishi, A Cuzzocrea, M Sharmin, D Towey, AKMJA Majumder, H Kashiwazaki, J-J Yang, M Takemoto, N Sakib, R Banno & SI Ahamed (eds), Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. Proceedings - International Computer Software and Applications Conference, vol. 2023-June, IEEE Computer Society, pp. 1870-1875, 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023, Hybrid, Torino, Italy, 06/26/23. https://doi.org/10.1109/COMPSAC57700.2023.00290

WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems. / Casanova, Henri; Berney, Kyle; Chastel, Serge et al.
Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. ed. / Hossain Shahriar; Yuuichi Teranishi; Alfredo Cuzzocrea; Moushumi Sharmin; Dave Towey; AKM Jahangir Alam Majumder; Hiroki Kashiwazaki; Ji-Jiang Yang; Michiharu Takemoto; Nazmus Sakib; Ryohei Banno; Sheikh Iqbal Ahamed. IEEE Computer Society, 2023. p. 1870-1875 (Proceedings - International Computer Software and Applications Conference; Vol. 2023-June).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - WfCommons

T2 - 47th IEEE Annual Computers, Software, and Applications Conference, COMPSAC 2023

AU - Casanova, Henri

AU - Berney, Kyle

AU - Chastel, Serge

AU - Da Silva, Rafael Ferreira

PY - 2023

Y1 - 2023

N2 - Scientific workflows have become ubiquitous across scientific fields, and their execution methods and systems continue to be the subject of research and development. Most experimental evaluations of these workflows rely on workflow instances, which can be either real-world or synthetic, to ensure relevance to current application domains or explore hypothetical/future scenarios. The WfCommons project addresses this need by providing data and tools to support such evaluations. In this paper, we present an overview of WfCommons and describe two recent developments. Firstly, we introduce a workflow execution "tracer"for Nextflow, which significantly enhances the set of real-world instances available in WfCommons. Secondly, we describe a workflow instance "translator"that enables the execution of any real-world or synthetic WfCommons workflow instance using Dask. Our contributions aim to provide researchers and practitioners with more comprehensive resources for evaluating scientific workflows.

AB - Scientific workflows have become ubiquitous across scientific fields, and their execution methods and systems continue to be the subject of research and development. Most experimental evaluations of these workflows rely on workflow instances, which can be either real-world or synthetic, to ensure relevance to current application domains or explore hypothetical/future scenarios. The WfCommons project addresses this need by providing data and tools to support such evaluations. In this paper, we present an overview of WfCommons and describe two recent developments. Firstly, we introduce a workflow execution "tracer"for Nextflow, which significantly enhances the set of real-world instances available in WfCommons. Secondly, we describe a workflow instance "translator"that enables the execution of any real-world or synthetic WfCommons workflow instance using Dask. Our contributions aim to provide researchers and practitioners with more comprehensive resources for evaluating scientific workflows.

KW - Scientific workflows

KW - workflow instance collection

KW - workflow instance execution

UR - https://www.scopus.com/pages/publications/85168914824

U2 - 10.1109/COMPSAC57700.2023.00290

DO - 10.1109/COMPSAC57700.2023.00290

M3 - Conference contribution

AN - SCOPUS:85168914824

T3 - Proceedings - International Computer Software and Applications Conference

SP - 1870

EP - 1875

BT - Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023

A2 - Shahriar, Hossain

A2 - Teranishi, Yuuichi

A2 - Cuzzocrea, Alfredo

A2 - Sharmin, Moushumi

A2 - Towey, Dave

A2 - Majumder, AKM Jahangir Alam

A2 - Kashiwazaki, Hiroki

A2 - Yang, Ji-Jiang

A2 - Takemoto, Michiharu

A2 - Sakib, Nazmus

A2 - Banno, Ryohei

A2 - Ahamed, Sheikh Iqbal

PB - IEEE Computer Society

Y2 - 26 June 2023 through 30 June 2023

ER -

Casanova H, Berney K, Chastel S, Da Silva RF. WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems. In Shahriar H, Teranishi Y, Cuzzocrea A, Sharmin M, Towey D, Majumder AKMJA, Kashiwazaki H, Yang JJ, Takemoto M, Sakib N, Banno R, Ahamed SI, editors, Proceedings - 2023 IEEE 47th Annual Computers, Software, and Applications Conference, COMPSAC 2023. IEEE Computer Society. 2023. p. 1870-1875. (Proceedings - International Computer Software and Applications Conference). doi: 10.1109/COMPSAC57700.2023.00290

WfCommons: Data Collection and Runtime Experiments using Multiple Workflow Systems

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this