Serverless containers - Rising viable approach to scientific workflows

Krzysztof Burkat; Maciej Pawlik; Bartosz Balis; Maciej Malawski; Karan Vahi; Mats Rynge; Rafael Ferreira Da Silva; Ewa Deelman

doi:10.1109/eScience51609.2021.00014

Serverless containers - Rising viable approach to scientific workflows

Krzysztof Burkat, Maciej Pawlik, Bartosz Balis, Maciej Malawski, Karan Vahi, Mats Rynge, Rafael Ferreira Da Silva, Ewa Deelman

Data Lifecycle Technologies

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

16 Scopus citations

Abstract

The increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. New infrastructures facilitate an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating the capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For the experimental evaluation of our approach, we extended the HyperFlow engine to support these CaaS platforms, together with adapting four scientific workflows composed of several dozen to hundreds of tasks organized into a dependency graph. Studied applications are used to create cost-performance benchmarks and flow execution plots, delay, elasticity, and scalability measurements. Results show that serverless containers can be successfully utilized for running scientific workflows. Moreover, the results allow for gaining insight into the specific advantages and limits of the studied platforms.

Original language	English
Title of host publication	Proceedings - IEEE 17th International Conference on eScience, eScience 2021
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	40-49
Number of pages	10
ISBN (Electronic)	9781665403610
DOIs	https://doi.org/10.1109/eScience51609.2021.00014
State	Published - Sep 2021
Event	17th IEEE International Conference on eScience, eScience 2021 - Virtual, Online, Austria Duration: Sep 20 2021 → Sep 23 2021

Publication series

Name	Proceedings - IEEE 17th International Conference on eScience, eScience 2021

Conference

Conference	17th IEEE International Conference on eScience, eScience 2021
Country/Territory	Austria
City	Virtual, Online
Period	09/20/21 → 09/23/21

Funding

This work was supported by the National Science Centre, Poland, grant 2016/21/B/ST6/01497 and (partially) supported by the funds of the Polish Ministry of Education and Science assigned to AGH University of Science and Technology.

Keywords

Containers
Evaluation
Infrastructure
Scientific workflows
Serverless

Access to Document

10.1109/eScience51609.2021.00014

Cite this

Burkat, K., Pawlik, M., Balis, B., Malawski, M., Vahi, K., Rynge, M., Da Silva, R. F., & Deelman, E. (2021). Serverless containers - Rising viable approach to scientific workflows. In Proceedings - IEEE 17th International Conference on eScience, eScience 2021 (pp. 40-49). (Proceedings - IEEE 17th International Conference on eScience, eScience 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/eScience51609.2021.00014

@inproceedings{7acd5325a99d4ed396914c98766da3df,

title = "Serverless containers - Rising viable approach to scientific workflows",

abstract = "The increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. New infrastructures facilitate an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating the capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For the experimental evaluation of our approach, we extended the HyperFlow engine to support these CaaS platforms, together with adapting four scientific workflows composed of several dozen to hundreds of tasks organized into a dependency graph. Studied applications are used to create cost-performance benchmarks and flow execution plots, delay, elasticity, and scalability measurements. Results show that serverless containers can be successfully utilized for running scientific workflows. Moreover, the results allow for gaining insight into the specific advantages and limits of the studied platforms.",

keywords = "Containers, Evaluation, Infrastructure, Scientific workflows, Serverless",

author = "Krzysztof Burkat and Maciej Pawlik and Bartosz Balis and Maciej Malawski and Karan Vahi and Mats Rynge and {Da Silva}, {Rafael Ferreira} and Ewa Deelman",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 17th IEEE International Conference on eScience, eScience 2021 ; Conference date: 20-09-2021 Through 23-09-2021",

year = "2021",

month = sep,

doi = "10.1109/eScience51609.2021.00014",

language = "English",

series = "Proceedings - IEEE 17th International Conference on eScience, eScience 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "40--49",

booktitle = "Proceedings - IEEE 17th International Conference on eScience, eScience 2021",

}

Burkat, K, Pawlik, M, Balis, B, Malawski, M, Vahi, K, Rynge, M, Da Silva, RF & Deelman, E 2021, Serverless containers - Rising viable approach to scientific workflows. in Proceedings - IEEE 17th International Conference on eScience, eScience 2021. Proceedings - IEEE 17th International Conference on eScience, eScience 2021, Institute of Electrical and Electronics Engineers Inc., pp. 40-49, 17th IEEE International Conference on eScience, eScience 2021, Virtual, Online, Austria, 09/20/21. https://doi.org/10.1109/eScience51609.2021.00014

Serverless containers - Rising viable approach to scientific workflows. / Burkat, Krzysztof; Pawlik, Maciej; Balis, Bartosz et al.
Proceedings - IEEE 17th International Conference on eScience, eScience 2021. Institute of Electrical and Electronics Engineers Inc., 2021. p. 40-49 (Proceedings - IEEE 17th International Conference on eScience, eScience 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Serverless containers - Rising viable approach to scientific workflows

AU - Burkat, Krzysztof

AU - Pawlik, Maciej

AU - Balis, Bartosz

AU - Malawski, Maciej

AU - Vahi, Karan

AU - Rynge, Mats

AU - Da Silva, Rafael Ferreira

AU - Deelman, Ewa

PY - 2021/9

Y1 - 2021/9

N2 - The increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. New infrastructures facilitate an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating the capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For the experimental evaluation of our approach, we extended the HyperFlow engine to support these CaaS platforms, together with adapting four scientific workflows composed of several dozen to hundreds of tasks organized into a dependency graph. Studied applications are used to create cost-performance benchmarks and flow execution plots, delay, elasticity, and scalability measurements. Results show that serverless containers can be successfully utilized for running scientific workflows. Moreover, the results allow for gaining insight into the specific advantages and limits of the studied platforms.

AB - The increasing popularity of the serverless computing approach has led to the emergence of new cloud infrastructures working in Container-as-a-Service (CaaS) model like AWS Fargate, Google Cloud Run, or Azure Container Instances. New infrastructures facilitate an innovative approach to running cloud containers where developers are freed from managing underlying resources. In this paper, we focus on evaluating the capabilities of elastic containers and their usefulness for scientific computing in the scientific workflow paradigm using AWS Fargate and Google Cloud Run infrastructures. For the experimental evaluation of our approach, we extended the HyperFlow engine to support these CaaS platforms, together with adapting four scientific workflows composed of several dozen to hundreds of tasks organized into a dependency graph. Studied applications are used to create cost-performance benchmarks and flow execution plots, delay, elasticity, and scalability measurements. Results show that serverless containers can be successfully utilized for running scientific workflows. Moreover, the results allow for gaining insight into the specific advantages and limits of the studied platforms.

KW - Containers

KW - Evaluation

KW - Infrastructure

KW - Scientific workflows

KW - Serverless

UR - http://www.scopus.com/inward/record.url?scp=85119047231&partnerID=8YFLogxK

U2 - 10.1109/eScience51609.2021.00014

DO - 10.1109/eScience51609.2021.00014

M3 - Conference contribution

AN - SCOPUS:85119047231

T3 - Proceedings - IEEE 17th International Conference on eScience, eScience 2021

SP - 40

EP - 49

BT - Proceedings - IEEE 17th International Conference on eScience, eScience 2021

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 17th IEEE International Conference on eScience, eScience 2021

Y2 - 20 September 2021 through 23 September 2021

ER -

Burkat K, Pawlik M, Balis B, Malawski M, Vahi K, Rynge M et al. Serverless containers - Rising viable approach to scientific workflows. In Proceedings - IEEE 17th International Conference on eScience, eScience 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 40-49. (Proceedings - IEEE 17th International Conference on eScience, eScience 2021). doi: 10.1109/eScience51609.2021.00014

Serverless containers - Rising viable approach to scientific workflows

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this