Abstract
Modern large-scale scientific discovery requires multidisciplinary collaboration across diverse computing facilities, including High Performance Computing (HPC) machines and the Edge-to-Cloud continuum. Integrated data analysis plays a crucial role in scientific discovery, especially in the current AI era, by enabling Responsible AI development, FAIR, Reproducibility, and User Steering. However, the heterogeneous nature of science poses challenges such as dealing with multiple supporting tools, cross-facility environments, and efficient HPC execution. Building on data observability, adapter system design, and provenance, we propose MIDA: an approach for lightweight runtime Multi-workflow Integrated Data Analysis. MIDA defines data observability strategies and adaptability methods for various parallel systems and machine learning tools. With observability, it intercepts the dataflows in the background without requiring instrumentation while integrating domain, provenance, and telemetry data at runtime into a unified database ready for user steering queries. We conduct experiments showing end-to-end multi-workflow analysis integrating data from Dask and MLFlow in a real distributed deep learning use case for materials science that runs on multiple environments with up to 276 GPUs in parallel. We show near-zero overhead running up to 100,000 tasks on 1,680 CPU cores on the Summit supercomputer.
| Original language | English |
|---|---|
| Title of host publication | Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023 |
| Publisher | Institute of Electrical and Electronics Engineers Inc. |
| ISBN (Electronic) | 9798350322231 |
| DOIs | |
| State | Published - 2023 |
| Event | 19th IEEE International Conference on e-Science, e-Science 2023 - Limassol, Cyprus Duration: Oct 9 2023 → Oct 14 2023 |
Publication series
| Name | Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023 |
|---|
Conference
| Conference | 19th IEEE International Conference on e-Science, e-Science 2023 |
|---|---|
| Country/Territory | Cyprus |
| City | Limassol |
| Period | 10/9/23 → 10/14/23 |
Funding
Acknowledgements. To Ketan Maheshwari, Joshua Brown, and Mark Coletti (ORNL) for their help. This research used resources of the Oak Ridge Leadership Computing Facility at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. Support for DOI 10.13139/ORNLNCCS/1773704 dataset is provided by the U.S. Department of Energy, project Sm-BFO STEM under Contract DE-AC05-00OR22725. Project Sm-BFO STEM used resources of the OLCF at Oak Ridge National Laboratory, supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This manuscript has been authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, worldwide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).
Keywords
- Adaptability
- Cross-facility
- Dask
- Data Integration
- Data Observability
- Deep Learning
- Explainability
- Lineage
- Machine Learning
- Provenance
- Responsible AI
- Workflows