TY - GEN
T1 - Highly Interactive, Steered Scientific Workflows on HPC Systems
T2 - 34th International Conference on High Performance Computing, ISC High Performance 2019
AU - Ossyra, John R.
AU - Sedova, Ada
AU - Baker, Matthew B.
AU - Smith, Jeremy C.
N1 - Publisher Copyright:
© 2019, Springer Nature Switzerland AG.
PY - 2019
Y1 - 2019
N2 - Scientific workflows are becoming increasingly important in high performance computing (HPC) settings, as the feasibility and appeal of many simultaneous heterogeneous tasks increases with increasing hardware capabilities. Currently no HPC-based workflow platform supports a dynamically adaptable workflow with interactive steering and analysis at run-time. Furthermore, for most workflow programs, compute resources are fixed for a given instance, resulting in a possible waste of expensive allocation resources when tasks are spawned and killed. Here we describe the design and testing of a run-time-interactive, adaptable, steered workflow tool capable of executing thousands of parallel tasks without an MPI programming model, using a database management system to facilitate task management through multiple live connections. We find that on the Oak Ridge Leadership Computing Facility pre-exascale Summit supercomputer it is possible to launch and interactively steer workflows with thousands of simultaneous tasks with negligible latency. For the case of particle simulation and analysis tasks that run for minutes to hours, this paradigm offers the prospect of a robust and efficient means to perform simulation-space exploration with on-the-fly analysis and adaptation.
AB - Scientific workflows are becoming increasingly important in high performance computing (HPC) settings, as the feasibility and appeal of many simultaneous heterogeneous tasks increases with increasing hardware capabilities. Currently no HPC-based workflow platform supports a dynamically adaptable workflow with interactive steering and analysis at run-time. Furthermore, for most workflow programs, compute resources are fixed for a given instance, resulting in a possible waste of expensive allocation resources when tasks are spawned and killed. Here we describe the design and testing of a run-time-interactive, adaptable, steered workflow tool capable of executing thousands of parallel tasks without an MPI programming model, using a database management system to facilitate task management through multiple live connections. We find that on the Oak Ridge Leadership Computing Facility pre-exascale Summit supercomputer it is possible to launch and interactively steer workflows with thousands of simultaneous tasks with negligible latency. For the case of particle simulation and analysis tasks that run for minutes to hours, this paradigm offers the prospect of a robust and efficient means to perform simulation-space exploration with on-the-fly analysis and adaptation.
KW - Adaptable workflows
KW - External steering
KW - High performance computing
KW - Scientific workflows
UR - http://www.scopus.com/inward/record.url?scp=85076835438&partnerID=8YFLogxK
U2 - 10.1007/978-3-030-34356-9_39
DO - 10.1007/978-3-030-34356-9_39
M3 - Conference contribution
AN - SCOPUS:85076835438
SN - 9783030343552
T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
SP - 514
EP - 527
BT - High Performance Computing - ISC High Performance 2019 International Workshops, Revised Selected Papers
A2 - Weiland, Michèle
A2 - Juckeland, Guido
A2 - Alam, Sadaf
A2 - Jagode, Heike
PB - Springer
Y2 - 16 June 2019 through 20 June 2019
ER -