Enabling discovery data science through cross-facility workflows

K. B. Antypas; D. J. Bard; J. P. Blaschke; R. Shane Canon; Bjoern Enders; Mallikarjun Arjun Shankar; Suhas Somnath; Dale Stansberry; Thomas D. Uram; Sean R. Wilkinson

doi:10.1109/BigData52589.2021.9671421

Enabling discovery data science through cross-facility workflows

K. B. Antypas, D. J. Bard, J. P. Blaschke, R. Shane Canon, Bjoern Enders, Mallikarjun Arjun Shankar, Suhas Somnath, Dale Stansberry, Thomas D. Uram, Sean R. Wilkinson

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

11 Scopus citations

Abstract

Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.

Original language	English
Title of host publication	Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021
Editors	Yixin Chen, Heiko Ludwig, Yicheng Tu, Usama Fayyad, Xingquan Zhu, Xiaohua Tony Hu, Suren Byna, Xiong Liu, Jianping Zhang, Shirui Pan, Vagelis Papalexakis, Jianwu Wang, Alfredo Cuzzocrea, Carlos Ordonez
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	3671-3680
Number of pages	10
ISBN (Electronic)	9781665439022
DOIs	https://doi.org/10.1109/BigData52589.2021.9671421
State	Published - 2021
Event	2021 IEEE International Conference on Big Data, Big Data 2021 - Virtual, Online, United States Duration: Dec 15 2021 → Dec 18 2021

Publication series

Name	Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021

Conference

Conference	2021 IEEE International Conference on Big Data, Big Data 2021
Country/Territory	United States
City	Virtual, Online
Period	12/15/21 → 12/18/21

Keywords

big data science
containers
cross-facility workflows
data analysis
infrastructure
orchestration platforms
workflow portability

Access to Document

10.1109/BigData52589.2021.9671421

Cite this

Antypas, K. B., Bard, D. J., Blaschke, J. P., Shane Canon, R., Enders, B., Shankar, M. A., Somnath, S., Stansberry, D., Uram, T. D., & Wilkinson, S. R. (2021). Enabling discovery data science through cross-facility workflows. In Y. Chen, H. Ludwig, Y. Tu, U. Fayyad, X. Zhu, X. T. Hu, S. Byna, X. Liu, J. Zhang, S. Pan, V. Papalexakis, J. Wang, A. Cuzzocrea, & C. Ordonez (Eds.), Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021 (pp. 3671-3680). (Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData52589.2021.9671421

Antypas, K. B. ; Bard, D. J. ; Blaschke, J. P. et al. / Enabling discovery data science through cross-facility workflows. Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021. editor / Yixin Chen ; Heiko Ludwig ; Yicheng Tu ; Usama Fayyad ; Xingquan Zhu ; Xiaohua Tony Hu ; Suren Byna ; Xiong Liu ; Jianping Zhang ; Shirui Pan ; Vagelis Papalexakis ; Jianwu Wang ; Alfredo Cuzzocrea ; Carlos Ordonez. Institute of Electrical and Electronics Engineers Inc., 2021. pp. 3671-3680 (Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021).

@inproceedings{c0cea72bd22649d283f60e6045a3c009,

title = "Enabling discovery data science through cross-facility workflows",

abstract = "Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.",

keywords = "big data science, containers, cross-facility workflows, data analysis, infrastructure, orchestration platforms, workflow portability",

author = "Antypas, {K. B.} and Bard, {D. J.} and Blaschke, {J. P.} and {Shane Canon}, R. and Bjoern Enders and Shankar, {Mallikarjun Arjun} and Suhas Somnath and Dale Stansberry and Uram, {Thomas D.} and Wilkinson, {Sean R.}",

note = "Publisher Copyright: {\textcopyright} 2021 IEEE.; 2021 IEEE International Conference on Big Data, Big Data 2021 ; Conference date: 15-12-2021 Through 18-12-2021",

year = "2021",

doi = "10.1109/BigData52589.2021.9671421",

language = "English",

series = "Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "3671--3680",

editor = "Yixin Chen and Heiko Ludwig and Yicheng Tu and Usama Fayyad and Xingquan Zhu and Hu, {Xiaohua Tony} and Suren Byna and Xiong Liu and Jianping Zhang and Shirui Pan and Vagelis Papalexakis and Jianwu Wang and Alfredo Cuzzocrea and Carlos Ordonez",

booktitle = "Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021",

}

Antypas, KB, Bard, DJ, Blaschke, JP, Shane Canon, R, Enders, B, Shankar, MA, Somnath, S, Stansberry, D, Uram, TD & Wilkinson, SR 2021, Enabling discovery data science through cross-facility workflows. in Y Chen, H Ludwig, Y Tu, U Fayyad, X Zhu, XT Hu, S Byna, X Liu, J Zhang, S Pan, V Papalexakis, J Wang, A Cuzzocrea & C Ordonez (eds), Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021. Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021, Institute of Electrical and Electronics Engineers Inc., pp. 3671-3680, 2021 IEEE International Conference on Big Data, Big Data 2021, Virtual, Online, United States, 12/15/21. https://doi.org/10.1109/BigData52589.2021.9671421

Enabling discovery data science through cross-facility workflows. / Antypas, K. B.; Bard, D. J.; Blaschke, J. P. et al.
Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021. ed. / Yixin Chen; Heiko Ludwig; Yicheng Tu; Usama Fayyad; Xingquan Zhu; Xiaohua Tony Hu; Suren Byna; Xiong Liu; Jianping Zhang; Shirui Pan; Vagelis Papalexakis; Jianwu Wang; Alfredo Cuzzocrea; Carlos Ordonez. Institute of Electrical and Electronics Engineers Inc., 2021. p. 3671-3680 (Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Enabling discovery data science through cross-facility workflows

AU - Antypas, K. B.

AU - Bard, D. J.

AU - Blaschke, J. P.

AU - Shane Canon, R.

AU - Enders, Bjoern

AU - Shankar, Mallikarjun Arjun

AU - Somnath, Suhas

AU - Stansberry, Dale

AU - Uram, Thomas D.

AU - Wilkinson, Sean R.

PY - 2021

Y1 - 2021

N2 - Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.

AB - Experimental and observational instruments for scientific research (such as light sources, genome sequencers, accelerators, telescopes and electron microscopes) increasingly require High Performance Computing (HPC) scale capabilities for data analysis and workflow processing. Next-generation instruments are being deployed with higher resolutions and faster data capture rates, creating a big data crunch that cannot be handled by modest institutional computing resources. Often these big data analysis pipelines also require near real-time computing and have higher resilience requirements than the simulation and modeling workloads more traditionally seen at HPC centers. While some facilities have enabled workflows to run at a single HPC facility, there is a growing need to integrate capabilities across HPC facilities to enable cross-facility workflows, either to provide resilience to an experiment, increase analysis throughput capabilities, or to better match a workflow to a particular architecture. In this paper we describe the barriers to executing complex data analysis workflows across HPC facilities and propose an architectural design pattern for enabling scientific discovery using cross-facility workflows that includes orchestration services, application programming interfaces (APIs), data access and co-scheduling.

KW - big data science

KW - containers

KW - cross-facility workflows

KW - data analysis

KW - infrastructure

KW - orchestration platforms

KW - workflow portability

UR - http://www.scopus.com/inward/record.url?scp=85125355214&partnerID=8YFLogxK

U2 - 10.1109/BigData52589.2021.9671421

DO - 10.1109/BigData52589.2021.9671421

M3 - Conference contribution

AN - SCOPUS:85125355214

T3 - Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021

SP - 3671

EP - 3680

BT - Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021

A2 - Chen, Yixin

A2 - Ludwig, Heiko

A2 - Tu, Yicheng

A2 - Fayyad, Usama

A2 - Zhu, Xingquan

A2 - Hu, Xiaohua Tony

A2 - Byna, Suren

A2 - Liu, Xiong

A2 - Zhang, Jianping

A2 - Pan, Shirui

A2 - Papalexakis, Vagelis

A2 - Wang, Jianwu

A2 - Cuzzocrea, Alfredo

A2 - Ordonez, Carlos

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2021 IEEE International Conference on Big Data, Big Data 2021

Y2 - 15 December 2021 through 18 December 2021

ER -

Antypas KB, Bard DJ, Blaschke JP, Shane Canon R, Enders B, Shankar MA et al. Enabling discovery data science through cross-facility workflows. In Chen Y, Ludwig H, Tu Y, Fayyad U, Zhu X, Hu XT, Byna S, Liu X, Zhang J, Pan S, Papalexakis V, Wang J, Cuzzocrea A, Ordonez C, editors, Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021. Institute of Electrical and Electronics Engineers Inc. 2021. p. 3671-3680. (Proceedings - 2021 IEEE International Conference on Big Data, Big Data 2021). doi: 10.1109/BigData52589.2021.9671421

Enabling discovery data science through cross-facility workflows

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this