Custom execution environments with containers in pegasus-enabled scientific workflows

Karan Vahi; Michael Zink; Mats Rynge; George Papadimitriou; Duncan Brown; Rajiv Mayani; Rafael Ferreira Da Silva; Ewa Deelman; Anirban Mandal; Eric Lyons

doi:10.1109/eScience.2019.00039

Custom execution environments with containers in pegasus-enabled scientific workflows

Karan Vahi, Michael Zink, Mats Rynge, George Papadimitriou, Duncan Brown, Rajiv Mayani, Rafael Ferreira Da Silva, Ewa Deelman, Anirban Mandal, Eric Lyons

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

9 Scopus citations

Abstract

Science reproducibility is a cornerstone feature in scientific workflows. In most cases, this has been implemented as a way to exactly reproduce the computational steps taken to reach the final results. While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints and operating system name and versions. Though this may be sufficient for reproducibility in the short term, systems evolve and are replaced over time, breaking the underlying workflow reproducibility. A natural solution to this problem is containers, as they are well defined, have a lifetime independent of the underlying system, and can be user-controlled so that they can provide custom environments if needed. This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows. Further, this paper explores how the Pegasus Workflow Management System implements container support to address such challenges.

Original language	English
Title of host publication	Proceedings - IEEE 15th International Conference on eScience, eScience 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	281-290
Number of pages	10
ISBN (Electronic)	9781728124513
DOIs	https://doi.org/10.1109/eScience.2019.00039
State	Published - Sep 2019
Externally published	Yes
Event	15th IEEE International Conference on eScience, eScience 2019 - San Diego, United States Duration: Sep 24 2019 → Sep 27 2019

Publication series

Name	Proceedings - IEEE 15th International Conference on eScience, eScience 2019

Conference

Conference	15th IEEE International Conference on eScience, eScience 2019
Country/Territory	United States
City	San Diego
Period	09/24/19 → 09/27/19

Funding

ACKNOWLEDGMENTS This work is funded by NSF contract #1664162, “SI2-SSI: Pegasus: Automating Compute and Data Intensive Science”; and NSF contract #1826997, “CC* Integration: Delivering a Dynamic Network-Centric Platform for Data-Driven Science (DyNamo)”. Development of containerization in PyCBC was supported by NSF contract #1443047, “CIF21 DIBBs: Domain-Aware Management of Heterogeneous Workflows: Active Data Management for Gravitational-Wave Science Workflows.” DAB was supported in part by NSF contract #1748958 to the Kavli Institute for Theoretical Physics.

Keywords

Containers
Distributed computing
Docker
Pegasus
Reproducibility
Scientific workflows
Shifter
Singularity

Access to Document

10.1109/eScience.2019.00039

Cite this

Vahi, K., Zink, M., Rynge, M., Papadimitriou, G., Brown, D., Mayani, R., Ferreira Da Silva, R., Deelman, E., Mandal, A., & Lyons, E. (2019). Custom execution environments with containers in pegasus-enabled scientific workflows. In Proceedings - IEEE 15th International Conference on eScience, eScience 2019 (pp. 281-290). Article 9041700 (Proceedings - IEEE 15th International Conference on eScience, eScience 2019). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/eScience.2019.00039

@inproceedings{c46ec791ca274104a63b4e3d0b35392e,

title = "Custom execution environments with containers in pegasus-enabled scientific workflows",

abstract = "Science reproducibility is a cornerstone feature in scientific workflows. In most cases, this has been implemented as a way to exactly reproduce the computational steps taken to reach the final results. While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints and operating system name and versions. Though this may be sufficient for reproducibility in the short term, systems evolve and are replaced over time, breaking the underlying workflow reproducibility. A natural solution to this problem is containers, as they are well defined, have a lifetime independent of the underlying system, and can be user-controlled so that they can provide custom environments if needed. This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows. Further, this paper explores how the Pegasus Workflow Management System implements container support to address such challenges.",

keywords = "Containers, Distributed computing, Docker, Pegasus, Reproducibility, Scientific workflows, Shifter, Singularity",

author = "Karan Vahi and Michael Zink and Mats Rynge and George Papadimitriou and Duncan Brown and Rajiv Mayani and \{Ferreira Da Silva\}, Rafael and Ewa Deelman and Anirban Mandal and Eric Lyons",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 15th IEEE International Conference on eScience, eScience 2019 ; Conference date: 24-09-2019 Through 27-09-2019",

year = "2019",

month = sep,

doi = "10.1109/eScience.2019.00039",

language = "English",

series = "Proceedings - IEEE 15th International Conference on eScience, eScience 2019",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "281--290",

booktitle = "Proceedings - IEEE 15th International Conference on eScience, eScience 2019",

}

Vahi, K, Zink, M, Rynge, M, Papadimitriou, G, Brown, D, Mayani, R, Ferreira Da Silva, R, Deelman, E, Mandal, A & Lyons, E 2019, Custom execution environments with containers in pegasus-enabled scientific workflows. in Proceedings - IEEE 15th International Conference on eScience, eScience 2019., 9041700, Proceedings - IEEE 15th International Conference on eScience, eScience 2019, Institute of Electrical and Electronics Engineers Inc., pp. 281-290, 15th IEEE International Conference on eScience, eScience 2019, San Diego, United States, 09/24/19. https://doi.org/10.1109/eScience.2019.00039

Custom execution environments with containers in pegasus-enabled scientific workflows. / Vahi, Karan; Zink, Michael; Rynge, Mats et al.
Proceedings - IEEE 15th International Conference on eScience, eScience 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 281-290 9041700 (Proceedings - IEEE 15th International Conference on eScience, eScience 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Custom execution environments with containers in pegasus-enabled scientific workflows

AU - Vahi, Karan

AU - Zink, Michael

AU - Rynge, Mats

AU - Papadimitriou, George

AU - Brown, Duncan

AU - Mayani, Rajiv

AU - Ferreira Da Silva, Rafael

AU - Deelman, Ewa

AU - Mandal, Anirban

AU - Lyons, Eric

PY - 2019/9

Y1 - 2019/9

N2 - Science reproducibility is a cornerstone feature in scientific workflows. In most cases, this has been implemented as a way to exactly reproduce the computational steps taken to reach the final results. While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints and operating system name and versions. Though this may be sufficient for reproducibility in the short term, systems evolve and are replaced over time, breaking the underlying workflow reproducibility. A natural solution to this problem is containers, as they are well defined, have a lifetime independent of the underlying system, and can be user-controlled so that they can provide custom environments if needed. This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows. Further, this paper explores how the Pegasus Workflow Management System implements container support to address such challenges.

AB - Science reproducibility is a cornerstone feature in scientific workflows. In most cases, this has been implemented as a way to exactly reproduce the computational steps taken to reach the final results. While these steps are often completely described, including the input parameters, datasets, and codes, the environment in which these steps are executed is only described at a higher level with endpoints and operating system name and versions. Though this may be sufficient for reproducibility in the short term, systems evolve and are replaced over time, breaking the underlying workflow reproducibility. A natural solution to this problem is containers, as they are well defined, have a lifetime independent of the underlying system, and can be user-controlled so that they can provide custom environments if needed. This paper highlights some unique challenges that may arise when using containers in distributed scientific workflows. Further, this paper explores how the Pegasus Workflow Management System implements container support to address such challenges.

KW - Containers

KW - Distributed computing

KW - Docker

KW - Pegasus

KW - Reproducibility

KW - Scientific workflows

KW - Shifter

KW - Singularity

UR - https://www.scopus.com/pages/publications/85083168946

U2 - 10.1109/eScience.2019.00039

DO - 10.1109/eScience.2019.00039

M3 - Conference contribution

AN - SCOPUS:85083168946

T3 - Proceedings - IEEE 15th International Conference on eScience, eScience 2019

SP - 281

EP - 290

BT - Proceedings - IEEE 15th International Conference on eScience, eScience 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 15th IEEE International Conference on eScience, eScience 2019

Y2 - 24 September 2019 through 27 September 2019

ER -

Vahi K, Zink M, Rynge M, Papadimitriou G, Brown D, Mayani R et al. Custom execution environments with containers in pegasus-enabled scientific workflows. In Proceedings - IEEE 15th International Conference on eScience, eScience 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 281-290. 9041700. (Proceedings - IEEE 15th International Conference on eScience, eScience 2019). doi: 10.1109/eScience.2019.00039

Custom execution environments with containers in pegasus-enabled scientific workflows

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this