Exploration of Workflow Management Systems Emerging Features from Users Perspectives

Ryan Mitchell; Loyc Pottier; Steve Jacobs; Rafael Ferreira Da Silva; Mats Rynge; Karan Vahi; Ewa Deelman

doi:10.1109/BigData47090.2019.9005494

Exploration of Workflow Management Systems Emerging Features from Users Perspectives

Ryan Mitchell, Loyc Pottier, Steve Jacobs, Rafael Ferreira Da Silva, Mats Rynge, Karan Vahi, Ewa Deelman

Workflow and Ecosystem Services

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

27 Scopus citations

Abstract

There has been a recent emergence of new workflow applications focused on data analytics and machine learning. This emergence has precipitated a change in the workflow management landscape, causing the development of new dataoriented workflow management systems (WMSs) in addition to the earlier standard of task-oriented WMSs. In this paper, we summarize three general workflow use-cases and explore the unique requirements of each use-case in order to understand how WMSs from both workflow management models meet the requirements of each workflow use-case from the user's perspective. We analyze the applicability of the two models by carefully describing each model and by providing an examination of the different variations of WMSs that fall under the task driven model. To illustrate the strengths and weaknesses of each workflow management model, we summarize the key features of four production-ready WMSs: Pegasus, Makeflow, Apache Airflow, and Pachyderm. To deepen our analysis of the four WMSs examined in this paper,we implement three real-world use-cases to highlight the specifications and features of each WMS. We present our final assessment of each WMS after considering the following factors: usability, performance, ease of deployment, and relevance. The purpose of this work is to offer insights from the user's perspective into the research challenges that WMSs currently face due to the evolving workflow landscape.

Original language	English
Title of host publication	Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019
Editors	Chaitanya Baru, Jun Huan, Latifur Khan, Xiaohua Tony Hu, Ronay Ak, Yuanyuan Tian, Roger Barga, Carlo Zaniolo, Kisung Lee, Yanfang Fanny Ye
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	4537-4544
Number of pages	8
ISBN (Electronic)	9781728108582
DOIs	https://doi.org/10.1109/BigData47090.2019.9005494
State	Published - Dec 2019
Event	2019 IEEE International Conference on Big Data, Big Data 2019 - Los Angeles, United States Duration: Dec 9 2019 → Dec 12 2019

Publication series

Name	Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019

Conference

Conference	2019 IEEE International Conference on Big Data, Big Data 2019
Country/Territory	United States
City	Los Angeles
Period	12/9/19 → 12/12/19

Funding

Acknowledgments. This work is funded by NSF contract #1842042: “Pilot Study for a Cyberinfrastructure Center of Excellence”. The National Ecological Observatory Network is a program sponsored by the National Science Foundation and operated under cooperative agreement by Battelle Memorial Institute.

Keywords

Data-driven.
Scientific workflow
Task-driven
Workflow Management System

Access to Document

10.1109/BigData47090.2019.9005494

Cite this

Mitchell, R., Pottier, L., Jacobs, S., Silva, R. F. D., Rynge, M., Vahi, K., & Deelman, E. (2019). Exploration of Workflow Management Systems Emerging Features from Users Perspectives. In C. Baru, J. Huan, L. Khan, X. T. Hu, R. Ak, Y. Tian, R. Barga, C. Zaniolo, K. Lee, & Y. F. Ye (Eds.), Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019 (pp. 4537-4544). Article 9005494 (Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData47090.2019.9005494

Mitchell, Ryan ; Pottier, Loyc ; Jacobs, Steve et al. / Exploration of Workflow Management Systems Emerging Features from Users Perspectives. Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019. editor / Chaitanya Baru ; Jun Huan ; Latifur Khan ; Xiaohua Tony Hu ; Ronay Ak ; Yuanyuan Tian ; Roger Barga ; Carlo Zaniolo ; Kisung Lee ; Yanfang Fanny Ye. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 4537-4544 (Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019).

@inproceedings{5dea196161f84f80967a3292d5f58ef8,

title = "Exploration of Workflow Management Systems Emerging Features from Users Perspectives",

abstract = "There has been a recent emergence of new workflow applications focused on data analytics and machine learning. This emergence has precipitated a change in the workflow management landscape, causing the development of new dataoriented workflow management systems (WMSs) in addition to the earlier standard of task-oriented WMSs. In this paper, we summarize three general workflow use-cases and explore the unique requirements of each use-case in order to understand how WMSs from both workflow management models meet the requirements of each workflow use-case from the user's perspective. We analyze the applicability of the two models by carefully describing each model and by providing an examination of the different variations of WMSs that fall under the task driven model. To illustrate the strengths and weaknesses of each workflow management model, we summarize the key features of four production-ready WMSs: Pegasus, Makeflow, Apache Airflow, and Pachyderm. To deepen our analysis of the four WMSs examined in this paper,we implement three real-world use-cases to highlight the specifications and features of each WMS. We present our final assessment of each WMS after considering the following factors: usability, performance, ease of deployment, and relevance. The purpose of this work is to offer insights from the user's perspective into the research challenges that WMSs currently face due to the evolving workflow landscape.",

keywords = "Data-driven., Scientific workflow, Task-driven, Workflow Management System",

author = "Ryan Mitchell and Loyc Pottier and Steve Jacobs and Silva, {Rafael Ferreira Da} and Mats Rynge and Karan Vahi and Ewa Deelman",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 2019 IEEE International Conference on Big Data, Big Data 2019 ; Conference date: 09-12-2019 Through 12-12-2019",

year = "2019",

month = dec,

doi = "10.1109/BigData47090.2019.9005494",

language = "English",

series = "Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "4537--4544",

editor = "Chaitanya Baru and Jun Huan and Latifur Khan and Hu, {Xiaohua Tony} and Ronay Ak and Yuanyuan Tian and Roger Barga and Carlo Zaniolo and Kisung Lee and Ye, {Yanfang Fanny}",

booktitle = "Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019",

}

Mitchell, R, Pottier, L, Jacobs, S, Silva, RFD, Rynge, M, Vahi, K & Deelman, E 2019, Exploration of Workflow Management Systems Emerging Features from Users Perspectives. in C Baru, J Huan, L Khan, XT Hu, R Ak, Y Tian, R Barga, C Zaniolo, K Lee & YF Ye (eds), Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019., 9005494, Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019, Institute of Electrical and Electronics Engineers Inc., pp. 4537-4544, 2019 IEEE International Conference on Big Data, Big Data 2019, Los Angeles, United States, 12/9/19. https://doi.org/10.1109/BigData47090.2019.9005494

Exploration of Workflow Management Systems Emerging Features from Users Perspectives. / Mitchell, Ryan; Pottier, Loyc; Jacobs, Steve et al.
Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019. ed. / Chaitanya Baru; Jun Huan; Latifur Khan; Xiaohua Tony Hu; Ronay Ak; Yuanyuan Tian; Roger Barga; Carlo Zaniolo; Kisung Lee; Yanfang Fanny Ye. Institute of Electrical and Electronics Engineers Inc., 2019. p. 4537-4544 9005494 (Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Exploration of Workflow Management Systems Emerging Features from Users Perspectives

AU - Mitchell, Ryan

AU - Pottier, Loyc

AU - Jacobs, Steve

AU - Silva, Rafael Ferreira Da

AU - Rynge, Mats

AU - Vahi, Karan

AU - Deelman, Ewa

PY - 2019/12

Y1 - 2019/12

N2 - There has been a recent emergence of new workflow applications focused on data analytics and machine learning. This emergence has precipitated a change in the workflow management landscape, causing the development of new dataoriented workflow management systems (WMSs) in addition to the earlier standard of task-oriented WMSs. In this paper, we summarize three general workflow use-cases and explore the unique requirements of each use-case in order to understand how WMSs from both workflow management models meet the requirements of each workflow use-case from the user's perspective. We analyze the applicability of the two models by carefully describing each model and by providing an examination of the different variations of WMSs that fall under the task driven model. To illustrate the strengths and weaknesses of each workflow management model, we summarize the key features of four production-ready WMSs: Pegasus, Makeflow, Apache Airflow, and Pachyderm. To deepen our analysis of the four WMSs examined in this paper,we implement three real-world use-cases to highlight the specifications and features of each WMS. We present our final assessment of each WMS after considering the following factors: usability, performance, ease of deployment, and relevance. The purpose of this work is to offer insights from the user's perspective into the research challenges that WMSs currently face due to the evolving workflow landscape.

AB - There has been a recent emergence of new workflow applications focused on data analytics and machine learning. This emergence has precipitated a change in the workflow management landscape, causing the development of new dataoriented workflow management systems (WMSs) in addition to the earlier standard of task-oriented WMSs. In this paper, we summarize three general workflow use-cases and explore the unique requirements of each use-case in order to understand how WMSs from both workflow management models meet the requirements of each workflow use-case from the user's perspective. We analyze the applicability of the two models by carefully describing each model and by providing an examination of the different variations of WMSs that fall under the task driven model. To illustrate the strengths and weaknesses of each workflow management model, we summarize the key features of four production-ready WMSs: Pegasus, Makeflow, Apache Airflow, and Pachyderm. To deepen our analysis of the four WMSs examined in this paper,we implement three real-world use-cases to highlight the specifications and features of each WMS. We present our final assessment of each WMS after considering the following factors: usability, performance, ease of deployment, and relevance. The purpose of this work is to offer insights from the user's perspective into the research challenges that WMSs currently face due to the evolving workflow landscape.

KW - Data-driven.

KW - Scientific workflow

KW - Task-driven

KW - Workflow Management System

UR - http://www.scopus.com/inward/record.url?scp=85081366758&partnerID=8YFLogxK

U2 - 10.1109/BigData47090.2019.9005494

DO - 10.1109/BigData47090.2019.9005494

M3 - Conference contribution

AN - SCOPUS:85081366758

T3 - Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019

SP - 4537

EP - 4544

BT - Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019

A2 - Baru, Chaitanya

A2 - Huan, Jun

A2 - Khan, Latifur

A2 - Hu, Xiaohua Tony

A2 - Ak, Ronay

A2 - Tian, Yuanyuan

A2 - Barga, Roger

A2 - Zaniolo, Carlo

A2 - Lee, Kisung

A2 - Ye, Yanfang Fanny

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2019 IEEE International Conference on Big Data, Big Data 2019

Y2 - 9 December 2019 through 12 December 2019

ER -

Mitchell R, Pottier L, Jacobs S, Silva RFD, Rynge M, Vahi K et al. Exploration of Workflow Management Systems Emerging Features from Users Perspectives. In Baru C, Huan J, Khan L, Hu XT, Ak R, Tian Y, Barga R, Zaniolo C, Lee K, Ye YF, editors, Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 4537-4544. 9005494. (Proceedings - 2019 IEEE International Conference on Big Data, Big Data 2019). doi: 10.1109/BigData47090.2019.9005494

Exploration of Workflow Management Systems Emerging Features from Users Perspectives

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this