Abstract
Today’s computational, experimental, and observational sciences rely on computations that involve many related tasks. The success of a scientific mission often hinges on the computer automation of these workflows. In April 2015, the US Department of Energy (DOE) invited a diverse group of domain and computer scientists from national laboratories supported by the Office of Science, the National Nuclear Security Administration, from industry, and from academia to review the workflow requirements of DOE’s science and national security missions, to assess the current state of the art in science workflows, to understand the impact of emerging extreme-scale computing systems on those workflows, and to develop requirements for automated workflow management in future and existing environments. This article is a summary of the opinions of over 50 leading researchers attending this workshop. We highlight use cases, computing systems, workflow needs and conclude by summarizing the remaining challenges this community sees that inhibit large-scale scientific workflows from becoming a mainstream tool for extreme-scale science.
| Original language | English |
|---|---|
| Pages (from-to) | 159-175 |
| Number of pages | 17 |
| Journal | International Journal of High Performance Computing Applications |
| Volume | 32 |
| Issue number | 1 |
| DOIs | |
| State | Published - Jan 1 2018 |
Funding
Today’s computational, experimental, and observational sciences rely on computations that involve many related tasks. The success of a scientific mission often hinges on the computer automation of these workflows. In April 2015, the US Department of Energy (DOE) invited a diverse group of domain and computer scientists from national laboratories supported by the Office of Science, the National Nuclear Security Administration, from industry, and from academia to review the workflow requirements of DOE’s science and national security missions, to assess the current state of the art in science workflows, to understand the impact of emerging extreme-scale computing systems on those workflows, and to develop requirements for automated workflow management in future and existing environments. This article is a summary of the opinions of over 50 leading researchers attending this workshop. We highlight use cases, computing systems, workflow needs and conclude by summarizing the remaining challenges this community sees that inhibit large-scale scientific workflows from becoming a mainstream tool for extreme-scale science. The author(s) disclosed receipt of the following financial support for the research, authorship, and/or publication of this article: Argonne National Laboratory is managed by UChicago Argonne, LLC, for the U.S. Department of Energy Office of Science under contract DE-AC02-06CH11357. Sandia National Laboratories is a multi-program laboratory managed and operated by Sandia Corporation, a wholly owned subsidiary of Lockheed Martin Corporation, for the U.S. Department of Energy’s National Nuclear Security Administration under contract DE-AC04-94AL85000. This work was funded in part by the US Department of Energy under Contract #DESC0012636, ‘‘Panorama -Predictive Modeling and Diagnostic Monitoring of Extreme Science Workflows’’. Ilkay Altintas is the chief data science officer at SDSC, UC San Diego, where she is also the founder and director for the Workflows for Data Science Center of Excellence. Since joining SDSC in 2001, she has worked on different aspects of scientific workflows as a principal investigator across a wide range of cross-disciplinary NSF, DOE, NIH, and Moore Foundation projects. She is a co-initiator of the popular open-source Kepler Scientific Workflow System and the coauthor of publications related to computational data science and e-Sciences at the intersection of scientific workflows, provenance, distributed computing, bioin-formatics, observatory systems, conceptual data querying, and software modeling. Ilkay is the recipient of the first SDSC Pi Person of the Year in 2014, and the IEEE TCSC Award for Excellence in Scalable Computing for Early Career Researchers in 2015. Ilkay Altintas received her PhD degree from the University of Amsterdam in the Netherlands with an emphasis on provenance of workflow-driven collaborative science and she is currently an assistant research scientist at UC San Diego.
| Funders | Funder number |
|---|---|
| DOE’s science and national security missions | |
| Extreme Science Workflows | |
| Sandia Corporation | |
| U.S. Department Of Energy’s National Nuclear Security Administration | DE-AC04-94AL85000 |
| US Department of Energy | 0012636 |
| National Science Foundation | |
| National Institutes of Health | |
| U.S. Department of Energy | |
| Gordon and Betty Moore Foundation | |
| Lockheed Martin Corporation | |
| Office of Science | |
| National Nuclear Security Administration | |
| Argonne National Laboratory | |
| Sandia National Laboratories | |
| IEEE Foundation | |
| University of Chicago | |
| Savannah River Operations Office, U.S. Department of Energy | DE-AC02-06CH11357 |
| Horizon 2020 Framework Programme | 654241, 676559, 823830, 730976, 675728, 824087 |
Keywords
- Scientific workflows
- distributed computing
- extreme-scale computing
- in situ computing
- workflow models