A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications

G. Padovani; V. Anantharaj; L. Sacco; T. Kurihana; M. Bunino; K. Tsolaki; M. Girone; F. Antonio; C. Sopranzetti; M. Fronza; S. Fiore

doi:10.1109/SCW63240.2024.00253

A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications

G. Padovani
, V. Anantharaj
, L. Sacco
, T. Kurihana
, M. Bunino
, K. Tsolaki
, M. Girone
, F. Antonio
, C. Sopranzetti
, M. Fronza
, S. Fiore

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AI-based scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.

Original language	English
Title of host publication	Proceedings of SC 2024-W
Subtitle of host publication	Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	2024-2031
Number of pages	8
ISBN (Electronic)	9798350355543
DOIs	https://doi.org/10.1109/SCW63240.2024.00253
State	Published - 2024
Event	2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024 - Atlanta, United States Duration: Nov 17 2024 → Nov 22 2024

Publication series

Name	Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024
Country/Territory	United States
City	Atlanta
Period	11/17/24 → 11/22/24

Funding

This work was partially funded by the EU HE interTwin project (GA 101058386) and the EU HE Climateurope2 project (GA 101056933). Moreover, this work was partially funded under the NRRP, Mission 4 Component 2 Investment 1.4, by the European Union - NextGenerationEU (proj. nr. CN 00000013). This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05- 00OR22725.

Keywords

ML tasks
Provenance
multi-level
workflow

Access to Document

10.1109/SCW63240.2024.00253

Cite this

Padovani, G., Anantharaj, V., Sacco, L., Kurihana, T., Bunino, M., Tsolaki, K., Girone, M., Antonio, F., Sopranzetti, C., Fronza, M., & Fiore, S. (2024). A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications. In Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 2024-2031). (Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SCW63240.2024.00253

Padovani, G. ; Anantharaj, V. ; Sacco, L. et al. / A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications. Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2024. pp. 2024-2031 (Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{bb8975697ab74743885c6a44d5437bcd,

title = "A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications",

abstract = "Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AI-based scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.",

keywords = "ML tasks, Provenance, multi-level, workflow",

author = "G. Padovani and V. Anantharaj and L. Sacco and T. Kurihana and M. Bunino and K. Tsolaki and M. Girone and F. Antonio and C. Sopranzetti and M. Fronza and S. Fiore",

note = "Publisher Copyright: {\textcopyright} 2024 IEEE.; 2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024 ; Conference date: 17-11-2024 Through 22-11-2024",

year = "2024",

doi = "10.1109/SCW63240.2024.00253",

language = "English",

series = "Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "2024--2031",

booktitle = "Proceedings of SC 2024-W",

}

Padovani, G, Anantharaj, V, Sacco, L, Kurihana, T, Bunino, M, Tsolaki, K, Girone, M, Antonio, F, Sopranzetti, C, Fronza, M & Fiore, S 2024, A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications. in Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 2024-2031, 2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024, Atlanta, United States, 11/17/24. https://doi.org/10.1109/SCW63240.2024.00253

A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications. / Padovani, G.; Anantharaj, V.; Sacco, L. et al.
Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2024. p. 2024-2031 (Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications

AU - Padovani, G.

AU - Anantharaj, V.

AU - Sacco, L.

AU - Kurihana, T.

AU - Bunino, M.

AU - Tsolaki, K.

AU - Girone, M.

AU - Antonio, F.

AU - Sopranzetti, C.

AU - Fronza, M.

AU - Fiore, S.

PY - 2024

Y1 - 2024

N2 - Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AI-based scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.

AB - Scientific workflows and provenance are two faces of the same medal. While the former addresses the coordinated execution of multiple tasks over a set of computational resources, the latter relates to the historical record of data from its original sources. This paper highlights the importance of tracking multi-level provenance metadata in complex, AI-based scientific workflows as a way to (i) foster and (ii) expand documentation of experiments, (iii) enable reproducibility, (iv) address interpretability of the results, (v) facilitate performance bottlenecks diagnosis, and (vi) advance provenance exploration and analysis opportunities.

KW - ML tasks

KW - Provenance

KW - multi-level

KW - workflow

UR - https://www.scopus.com/pages/publications/85217153874

U2 - 10.1109/SCW63240.2024.00253

DO - 10.1109/SCW63240.2024.00253

M3 - Conference contribution

AN - SCOPUS:85217153874

T3 - Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 2024

EP - 2031

BT - Proceedings of SC 2024-W

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2024 Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis, SC Workshops 2024

Y2 - 17 November 2024 through 22 November 2024

ER -

Padovani G, Anantharaj V, Sacco L, Kurihana T, Bunino M, Tsolaki K et al. A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications. In Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2024. p. 2024-2031. (Proceedings of SC 2024-W: Workshops of the International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/SCW63240.2024.00253

A software ecosystem for multi-level provenance management in large-scale scientific workflows for AI applications

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this