Software architectures to integrate workflow engines in science gateways

Tristan Glatard; Marc Étienne Rousseau; Sorina Camarasu-Pop; Reza Adalat; Natacha Beck; Samir Das; Rafael Ferreira da Silva; Najmeh Khalili-Mahani; Vladimir Korkhov; Pierre Olivier Quirion; Pierre Rioux; Sílvia D. Olabarriaga; Pierre Bellec; Alan C. Evans

doi:10.1016/j.future.2017.01.005

Software architectures to integrate workflow engines in science gateways

Tristan Glatard, Marc Étienne Rousseau, Sorina Camarasu-Pop, Reza Adalat, Natacha Beck, Samir Das, Rafael Ferreira da Silva, Najmeh Khalili-Mahani, Vladimir Korkhov, Pierre Olivier Quirion, Pierre Rioux, Sílvia D. Olabarriaga, Pierre Bellec, Alan C. Evans

Workflow and Ecosystem Services

Research output: Contribution to journal › Article › peer-review

10 Scopus citations

Abstract

Science gateways often rely on workflow engines to execute applications on distributed infrastructures. We investigate six software architectures commonly used to integrate workflow engines into science gateways. In tight integration, the workflow engine shares software components with the science gateway. In service invocation, the engine is isolated and invoked through a specific software interface. In task encapsulation, the engine is wrapped as a computing task executed on the infrastructure. In the pool model, the engine is bundled in an agent that connects to a central pool to fetch and execute workflows. In nested workflows, the engine is integrated as a child process of another engine. In workflow conversion, the engine is integrated through workflow language conversion. We describe and evaluate these architectures with metrics for assessment of integration complexity, robustness, extensibility, scalability and functionality. Tight integration and task encapsulation are the easiest to integrate and the most robust. Extensibility is equivalent in most architectures. The pool model is the most scalable one and meta-workflows are only available in nested workflows and workflow conversion. These results provide insights for science gateway architects and developers.

Original language	English
Pages (from-to)	239-255
Number of pages	17
Journal	Future Generation Computer Systems
Volume	75
DOIs	https://doi.org/10.1016/j.future.2017.01.005
State	Published - Oct 2017

Funding

We thank the anonymous reviewers for the thorough reviews and useful comments that greatly contributed to improve the quality of this paper. This work has been made possible with the support of the Irving Ludmer Family Foundation and the Ludmer Centre for Neuroinformatics and Mental Health. The integration between PSOM and CBRAIN was supported by a Brain Canada Platform Support Grant, as well as the Canadian Consortium on Neurodegeneration in Aging (CCNA), through a grant from the Canadian Institute of Health Research and funding from several partners. This work is in the scope of the LABEX PRIMES (ANR-11- LABX-0063) of Universit\u00E9 de Lyon, within the program \u201CInvestissements d'Avenir\u201D (ANR-11-IDEX-0007) operated by the French National Research Agency (ANR). This work also falls into the scope of the scientific topics of the French National Grid Institute (IdG). The VIP team thanks the site administrators of the European Grid Initiative and the GGUS support for their help related to the VIP platform. The CBRAIN team is grateful for the computing cycles, storage, and support obtained from Compute Canada (https://computecanada.ca) and platform development program from CANARIE (http://www.canarie.ca/). We also acknowledge the Dutch national e-Infrastructure with the support of SURF Cooperative, the Dutch national program COMMIT/ and the High Performance Computing and Networking (HPCN) Fund of the University of Amsterdam for their support to the science gateway activities at the AMC. We are also grateful to the financial support provided by FP7 E-INFRASTRUCTURE program for financial support to SCI-BUS, SHIWA and ER-flow projects.

Keywords

Science gateways
Software architectures
Workflow engines

Access to Document

10.1016/j.future.2017.01.005

Cite this

Glatard, T., Rousseau, M. É., Camarasu-Pop, S., Adalat, R., Beck, N., Das, S., da Silva, R. F., Khalili-Mahani, N., Korkhov, V., Quirion, P. O., Rioux, P., Olabarriaga, S. D., Bellec, P., & Evans, A. C. (2017). Software architectures to integrate workflow engines in science gateways. Future Generation Computer Systems, 75, 239-255. https://doi.org/10.1016/j.future.2017.01.005

@article{8ea0a47970bc4c61836d5d806df01e24,

title = "Software architectures to integrate workflow engines in science gateways",

abstract = "Science gateways often rely on workflow engines to execute applications on distributed infrastructures. We investigate six software architectures commonly used to integrate workflow engines into science gateways. In tight integration, the workflow engine shares software components with the science gateway. In service invocation, the engine is isolated and invoked through a specific software interface. In task encapsulation, the engine is wrapped as a computing task executed on the infrastructure. In the pool model, the engine is bundled in an agent that connects to a central pool to fetch and execute workflows. In nested workflows, the engine is integrated as a child process of another engine. In workflow conversion, the engine is integrated through workflow language conversion. We describe and evaluate these architectures with metrics for assessment of integration complexity, robustness, extensibility, scalability and functionality. Tight integration and task encapsulation are the easiest to integrate and the most robust. Extensibility is equivalent in most architectures. The pool model is the most scalable one and meta-workflows are only available in nested workflows and workflow conversion. These results provide insights for science gateway architects and developers.",

keywords = "Science gateways, Software architectures, Workflow engines",

author = "Tristan Glatard and Rousseau, {Marc {\'E}tienne} and Sorina Camarasu-Pop and Reza Adalat and Natacha Beck and Samir Das and {da Silva}, {Rafael Ferreira} and Najmeh Khalili-Mahani and Vladimir Korkhov and Quirion, {Pierre Olivier} and Pierre Rioux and Olabarriaga, {S{\'i}lvia D.} and Pierre Bellec and Evans, {Alan C.}",

note = "Publisher Copyright: {\textcopyright} 2017 The Author(s)",

year = "2017",

month = oct,

doi = "10.1016/j.future.2017.01.005",

language = "English",

volume = "75",

pages = "239--255",

journal = "Future Generation Computer Systems",

issn = "0167-739X",

publisher = "Elsevier",

}

TY - JOUR

T1 - Software architectures to integrate workflow engines in science gateways

AU - Glatard, Tristan

AU - Rousseau, Marc Étienne

AU - Camarasu-Pop, Sorina

AU - Adalat, Reza

AU - Beck, Natacha

AU - Das, Samir

AU - da Silva, Rafael Ferreira

AU - Khalili-Mahani, Najmeh

AU - Korkhov, Vladimir

AU - Quirion, Pierre Olivier

AU - Rioux, Pierre

AU - Olabarriaga, Sílvia D.

AU - Bellec, Pierre

AU - Evans, Alan C.

PY - 2017/10

Y1 - 2017/10

N2 - Science gateways often rely on workflow engines to execute applications on distributed infrastructures. We investigate six software architectures commonly used to integrate workflow engines into science gateways. In tight integration, the workflow engine shares software components with the science gateway. In service invocation, the engine is isolated and invoked through a specific software interface. In task encapsulation, the engine is wrapped as a computing task executed on the infrastructure. In the pool model, the engine is bundled in an agent that connects to a central pool to fetch and execute workflows. In nested workflows, the engine is integrated as a child process of another engine. In workflow conversion, the engine is integrated through workflow language conversion. We describe and evaluate these architectures with metrics for assessment of integration complexity, robustness, extensibility, scalability and functionality. Tight integration and task encapsulation are the easiest to integrate and the most robust. Extensibility is equivalent in most architectures. The pool model is the most scalable one and meta-workflows are only available in nested workflows and workflow conversion. These results provide insights for science gateway architects and developers.

AB - Science gateways often rely on workflow engines to execute applications on distributed infrastructures. We investigate six software architectures commonly used to integrate workflow engines into science gateways. In tight integration, the workflow engine shares software components with the science gateway. In service invocation, the engine is isolated and invoked through a specific software interface. In task encapsulation, the engine is wrapped as a computing task executed on the infrastructure. In the pool model, the engine is bundled in an agent that connects to a central pool to fetch and execute workflows. In nested workflows, the engine is integrated as a child process of another engine. In workflow conversion, the engine is integrated through workflow language conversion. We describe and evaluate these architectures with metrics for assessment of integration complexity, robustness, extensibility, scalability and functionality. Tight integration and task encapsulation are the easiest to integrate and the most robust. Extensibility is equivalent in most architectures. The pool model is the most scalable one and meta-workflows are only available in nested workflows and workflow conversion. These results provide insights for science gateway architects and developers.

KW - Science gateways

KW - Software architectures

KW - Workflow engines

UR - http://www.scopus.com/inward/record.url?scp=85011356088&partnerID=8YFLogxK

U2 - 10.1016/j.future.2017.01.005

DO - 10.1016/j.future.2017.01.005

M3 - Article

AN - SCOPUS:85011356088

SN - 0167-739X

VL - 75

SP - 239

EP - 255

JO - Future Generation Computer Systems

JF - Future Generation Computer Systems

ER -

Software architectures to integrate workflow engines in science gateways

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this