SWARM: Reimagining scientific workflow management systems in a distributed world

Prasanna Balaprakash; Krishnan Raghavan; Franck Cappello; Ewa Deelman; Anirban Mandal; Hongwei Jin; Imtiaz Mahmud; Komal Thareja; Shixun Wu; Pawel Zuk; Mariam Kiran; Zizhong Chen; Sheng Di; Kesheng Wu

doi:10.1177/10943420251339317

SWARM: Reimagining scientific workflow management systems in a distributed world

Prasanna Balaprakash, Krishnan Raghavan, Franck Cappello, Ewa Deelman, Anirban Mandal, Hongwei Jin, Imtiaz Mahmud, Komal Thareja, Shixun Wu, Pawel Zuk, Mariam Kiran, Zizhong Chen, Sheng Di, Kesheng Wu

Research output: Contribution to journal › Article › peer-review

Abstract

Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

Original language	English
Article number	10943420251339317
Journal	International Journal of High Performance Computing Applications
DOIs	https://doi.org/10.1177/10943420251339317
State	Accepted/In press - 2025

Funding

The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: The SWARM project is supported by the Department of Energy Award #DE-SC0024387.

Keywords

agentic AI
consensus algorithms
large language models
network overlays
resilience
Swarm intelligence
workflow systems

Access to Document

10.1177/10943420251339317

Cite this

Balaprakash, P., Raghavan, K., Cappello, F., Deelman, E., Mandal, A., Jin, H., Mahmud, I., Thareja, K., Wu, S., Zuk, P., Kiran, M., Chen, Z., Di, S., & Wu, K. (Accepted/In press). SWARM: Reimagining scientific workflow management systems in a distributed world. International Journal of High Performance Computing Applications, Article 10943420251339317. https://doi.org/10.1177/10943420251339317

@article{20f70cbee6f943099c56d47634c5a5c3,

title = "SWARM: Reimagining scientific workflow management systems in a distributed world",

abstract = "Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS{\textquoteright}s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.",

keywords = "agentic AI, consensus algorithms, large language models, network overlays, resilience, Swarm intelligence, workflow systems",

author = "Prasanna Balaprakash and Krishnan Raghavan and Franck Cappello and Ewa Deelman and Anirban Mandal and Hongwei Jin and Imtiaz Mahmud and Komal Thareja and Shixun Wu and Pawel Zuk and Mariam Kiran and Zizhong Chen and Sheng Di and Kesheng Wu",

note = "Publisher Copyright: {\textcopyright} The Author(s) 2025.",

year = "2025",

doi = "10.1177/10943420251339317",

language = "English",

journal = "International Journal of High Performance Computing Applications",

issn = "1094-3420",

publisher = "SAGE Publications",

}

TY - JOUR

T1 - SWARM

T2 - Reimagining scientific workflow management systems in a distributed world

AU - Balaprakash, Prasanna

AU - Raghavan, Krishnan

AU - Cappello, Franck

AU - Deelman, Ewa

AU - Mandal, Anirban

AU - Jin, Hongwei

AU - Mahmud, Imtiaz

AU - Thareja, Komal

AU - Wu, Shixun

AU - Zuk, Pawel

AU - Kiran, Mariam

AU - Chen, Zizhong

AU - Di, Sheng

AU - Wu, Kesheng

PY - 2025

Y1 - 2025

N2 - Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

AB - Modern scientific workflows process massive amounts of data from diverse instruments and sensors, leveraging geographically distributed, heterogeneous compute and storage resources—from leadership-class systems to edge devices—connected by high-performance networks. The diversity of resources introduces challenges in harnessing their full potential, with resilience issues arising across applications, system software, networks, storage, and hardware. Today, workflow management systems (WMS) coordinate the execution of computation and data management tasks across target resources. However, WMS’s centralized nature makes them vulnerable to faults and scalability issues that may result in failures of entire computational campaigns. This paper introduces a novel agentic framework for workflow management, fully distributing and decentralizing the WMS functions and modeling them as swarm intelligence agents infused with advanced artificial intelligence solutions and traditional distributed computing algorithms that can make coordinated decisions in the presence of failures of the underlying cyberinfrastructure.

KW - agentic AI

KW - consensus algorithms

KW - large language models

KW - network overlays

KW - resilience

KW - Swarm intelligence

KW - workflow systems

UR - http://www.scopus.com/inward/record.url?scp=105005578173&partnerID=8YFLogxK

U2 - 10.1177/10943420251339317

DO - 10.1177/10943420251339317

M3 - Article

AN - SCOPUS:105005578173

SN - 1094-3420

JO - International Journal of High Performance Computing Applications

JF - International Journal of High Performance Computing Applications

M1 - 10943420251339317

ER -

SWARM: Reimagining scientific workflow management systems in a distributed world

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this