The Pegasus workflow management system: Translational computer science in practice

Ewa Deelman, Rafael Ferreira da Silva, Karan Vahi, Mats Rynge, Rajiv Mayani, Ryan Tanaka, Wendy Whitcup, Miron Livny

Research output: Contribution to journalArticlepeer-review

15 Scopus citations

Abstract

Translational research (TR) has been extensively used in the health science domain, where results from laboratory research are translated to human studies and where evidence-based practices are adopted in real-world settings to reach broad communities. In computer science, much research stops at the result publication and dissemination stage without moving to the evaluation in real settings at scale and feeding the gained knowledge back to research. Additionally, there is a lack of steady funding and incentives to broadly promote translational computer science (TCS) in practice. In this paper, we present how, throughout its lifespan, the Pegasus workflow management system project has incorporated the principles of translational computer science. We report on our experience on building a strong, long-term engagement with a broad range of science communities to establish mutually beneficial relationships between the core R&D team and these communities.

Original languageEnglish
Article number101200
JournalJournal of Computational Science
Volume52
DOIs
StatePublished - May 2021
Externally publishedYes

Funding

Although the benefits of TR have been well-acknowledged in the health sciences domain (both by scientists and funding bodies), the traditional research workflow in computer science (CS) still does not have well-defined best practices. Additionally, there is a lack of support in CS for embracing TR as part of the critical path of the research development process [5] . On the other hand, as CS permeates not only traditional physical sciences, but also social sciences, humanities, and in fact every aspect of our lives, some CS researchers see the value of examining the entire innovation cycle from conceptualization and initial research to testing at scale in real environments to broad community adoption. In this context, this paper examines translational computer science (TCS) as defined by Abramson and Parashar [5] , and describes how this process of innovation has driven the development and adoption of the Pegasus workflow management system [6,7] , and what were the challenges that the project faced over the years. Pegasus came out of an NSF-funded project that aimed to explore the concept of virtual data in science [8] . That grant specifically funded interdisciplinary research in CS and astronomy/physics. As a result, Pegasus was born out of our belief that the value of our work lay not only in the novel computer science algorithms but also in the software that leveraged these algorithms to advance domain science. Thus, the translational aspects of CS grew organically to some degree. In CS, research projects are traditionally funded by governmental agencies that typically sponsor projects within an average time frame of 2–3 years. Although this allows research teams to produce reasonable outcomes that may enrich the state-of-the-art in the researched field or produce software elements to fulfill community pressing needs, there is no room for translating research outcomes to practice (if it is even considered at all). To overcome this challenge, the Pegasus project has been supported by a mixture of awards that target specific software development (NSF CI-focused programs), and fundamental and applied research (including NSF, DOE, NIH, and DARPA). While fundamental CS research is mostly focused on elemental aspects of novel algorithms and emerging technologies, applied research targets the distributed and heterogeneous aspect of Pegasus’ applications, which drives new requirements – note that in these awards Pegasus is partly supported as a tool for enabling efficient distributed processing. Despite our ability to secure funding from different sources for Pegasus R&D along these nearly two decades, we argue that continued, steady funding (preferably from fewer sources, as expectations may significantly shift from different agencies) is vital to have the time to bring back the real world experiences to the research and the software. For instance, often times, meaningful “field results” and scientific breakthroughs take time to come about (e.g., confirmation of the existence of gravitational waves [12] occurred 15 years after the initial collaboration between Pegasus and LIGO was established), and as a result the value of TCS is not evident in the near term or within typical funding cycles.

FundersFunder number
fundamental and applied research
National Science Foundation1664162
National Institutes of Health
U.S. Department of Energy
Defense Advanced Research Projects Agency

    Keywords

    • Large-scale distributed computing
    • Scientific workflows
    • Translational computer science
    • Translational research
    • Workflow management systems

    Fingerprint

    Dive into the research topics of 'The Pegasus workflow management system: Translational computer science in practice'. Together they form a unique fingerprint.

    Cite this