Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere

Jieyang Chen; Qiang Guan; Xin Liang; Paul Bryant; Patricia Grubel; Allen McPherson; Li Ta Lo; Timothy Randles; Zizhong Chen; James Paul Ahrens

doi:10.1109/BigData.2018.8622572

Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere

Jieyang Chen, Qiang Guan, Xin Liang, Paul Bryant, Patricia Grubel, Allen McPherson, Li Ta Lo, Timothy Randles, Zizhong Chen, James Paul Ahrens

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Variations in High Performance Computing (HPC) system software configurations mean that applications are typically configured and built for specific HPC environments. Building applications can require a significant investment of time and effort for application users and requires application users to have additional technical knowledge. Linux container technologies such as Docker and Charliecloud bring great benefits to the application development, build and deployment processes. While cloud platforms already widely support containers, HPC systems still have non-uniform support of container technologies. In this work, we propose a unified runtime framework - Build and Execution Environment (BEE) across both HPC and cloud platforms that allows users to run their containerized HPC applications across all supported platforms without modification. We design four BEE backends for four different classes of HPC or cloud platform so that together they cover the majority of mainstream computing platforms for HPC users. Evaluations show that BEE provides an easy-to-use unified user interface, execution environment, and comparable performance.

Original language	English
Title of host publication	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
Editors	Naoki Abe, Huan Liu, Calton Pu, Xiaohua Hu, Nesreen Ahmed, Mu Qiao, Yang Song, Donald Kossmann, Bing Liu, Kisung Lee, Jiliang Tang, Jingrui He, Jeffrey Saltz
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1737-1746
Number of pages	10
ISBN (Electronic)	9781538650356
DOIs	https://doi.org/10.1109/BigData.2018.8622572
State	Published - Jul 2 2018
Externally published	Yes
Event	2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States Duration: Dec 10 2018 → Dec 13 2018

Publication series

Name	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

Conference

Conference	2018 IEEE International Conference on Big Data, Big Data 2018
Country/Territory	United States
City	Seattle
Period	12/10/18 → 12/13/18

Funding

This work was funded by the the US Government contract DE-AC52-06NA25396 for Los Alamos National Laboratory, operated by Los Alamos National Security, LLC, for the US Department of Energy. This work was also supported by NSF Award No. 1513201. Results presented in this paper were obtained using the Chameleon Cloud sponsored by the National Science Foundation. The publication has been assigned the LANL identifier LA-UR-18-27912.

Keywords

cloud computing
container
high performance computing

Access to Document

10.1109/BigData.2018.8622572

Cite this

Chen, J., Guan, Q., Liang, X., Bryant, P., Grubel, P., McPherson, A., Lo, L. T., Randles, T., Chen, Z., & Ahrens, J. P. (2018). Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere. In N. Abe, H. Liu, C. Pu, X. Hu, N. Ahmed, M. Qiao, Y. Song, D. Kossmann, B. Liu, K. Lee, J. Tang, J. He, & J. Saltz (Eds.), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018 (pp. 1737-1746). Article 8622572 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2018.8622572

Chen, Jieyang ; Guan, Qiang ; Liang, Xin et al. / Build and Execution Environment (BEE) : An Encapsulated Environment Enabling HPC Applications Running Everywhere. Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. editor / Naoki Abe ; Huan Liu ; Calton Pu ; Xiaohua Hu ; Nesreen Ahmed ; Mu Qiao ; Yang Song ; Donald Kossmann ; Bing Liu ; Kisung Lee ; Jiliang Tang ; Jingrui He ; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 1737-1746 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

@inproceedings{9e89ec48b5fe43ed88363505112f32fd,

title = "Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere",

abstract = "Variations in High Performance Computing (HPC) system software configurations mean that applications are typically configured and built for specific HPC environments. Building applications can require a significant investment of time and effort for application users and requires application users to have additional technical knowledge. Linux container technologies such as Docker and Charliecloud bring great benefits to the application development, build and deployment processes. While cloud platforms already widely support containers, HPC systems still have non-uniform support of container technologies. In this work, we propose a unified runtime framework - Build and Execution Environment (BEE) across both HPC and cloud platforms that allows users to run their containerized HPC applications across all supported platforms without modification. We design four BEE backends for four different classes of HPC or cloud platform so that together they cover the majority of mainstream computing platforms for HPC users. Evaluations show that BEE provides an easy-to-use unified user interface, execution environment, and comparable performance.",

keywords = "cloud computing, container, high performance computing",

author = "Jieyang Chen and Qiang Guan and Xin Liang and Paul Bryant and Patricia Grubel and Allen McPherson and Lo, {Li Ta} and Timothy Randles and Zizhong Chen and Ahrens, {James Paul}",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 IEEE International Conference on Big Data, Big Data 2018 ; Conference date: 10-12-2018 Through 13-12-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/BigData.2018.8622572",

language = "English",

series = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1737--1746",

editor = "Naoki Abe and Huan Liu and Calton Pu and Xiaohua Hu and Nesreen Ahmed and Mu Qiao and Yang Song and Donald Kossmann and Bing Liu and Kisung Lee and Jiliang Tang and Jingrui He and Jeffrey Saltz",

booktitle = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

}

Chen, J, Guan, Q, Liang, X, Bryant, P, Grubel, P, McPherson, A, Lo, LT, Randles, T, Chen, Z & Ahrens, JP 2018, Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere. in N Abe, H Liu, C Pu, X Hu, N Ahmed, M Qiao, Y Song, D Kossmann, B Liu, K Lee, J Tang, J He & J Saltz (eds), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018., 8622572, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, Institute of Electrical and Electronics Engineers Inc., pp. 1737-1746, 2018 IEEE International Conference on Big Data, Big Data 2018, Seattle, United States, 12/10/18. https://doi.org/10.1109/BigData.2018.8622572

Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere. / Chen, Jieyang; Guan, Qiang; Liang, Xin et al.
Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. ed. / Naoki Abe; Huan Liu; Calton Pu; Xiaohua Hu; Nesreen Ahmed; Mu Qiao; Yang Song; Donald Kossmann; Bing Liu; Kisung Lee; Jiliang Tang; Jingrui He; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1737-1746 8622572 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Build and Execution Environment (BEE)

T2 - 2018 IEEE International Conference on Big Data, Big Data 2018

AU - Chen, Jieyang

AU - Guan, Qiang

AU - Liang, Xin

AU - Bryant, Paul

AU - Grubel, Patricia

AU - McPherson, Allen

AU - Lo, Li Ta

AU - Randles, Timothy

AU - Chen, Zizhong

AU - Ahrens, James Paul

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Variations in High Performance Computing (HPC) system software configurations mean that applications are typically configured and built for specific HPC environments. Building applications can require a significant investment of time and effort for application users and requires application users to have additional technical knowledge. Linux container technologies such as Docker and Charliecloud bring great benefits to the application development, build and deployment processes. While cloud platforms already widely support containers, HPC systems still have non-uniform support of container technologies. In this work, we propose a unified runtime framework - Build and Execution Environment (BEE) across both HPC and cloud platforms that allows users to run their containerized HPC applications across all supported platforms without modification. We design four BEE backends for four different classes of HPC or cloud platform so that together they cover the majority of mainstream computing platforms for HPC users. Evaluations show that BEE provides an easy-to-use unified user interface, execution environment, and comparable performance.

AB - Variations in High Performance Computing (HPC) system software configurations mean that applications are typically configured and built for specific HPC environments. Building applications can require a significant investment of time and effort for application users and requires application users to have additional technical knowledge. Linux container technologies such as Docker and Charliecloud bring great benefits to the application development, build and deployment processes. While cloud platforms already widely support containers, HPC systems still have non-uniform support of container technologies. In this work, we propose a unified runtime framework - Build and Execution Environment (BEE) across both HPC and cloud platforms that allows users to run their containerized HPC applications across all supported platforms without modification. We design four BEE backends for four different classes of HPC or cloud platform so that together they cover the majority of mainstream computing platforms for HPC users. Evaluations show that BEE provides an easy-to-use unified user interface, execution environment, and comparable performance.

KW - cloud computing

KW - container

KW - high performance computing

UR - http://www.scopus.com/inward/record.url?scp=85062637378&partnerID=8YFLogxK

U2 - 10.1109/BigData.2018.8622572

DO - 10.1109/BigData.2018.8622572

M3 - Conference contribution

AN - SCOPUS:85062637378

T3 - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

SP - 1737

EP - 1746

BT - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

A2 - Abe, Naoki

A2 - Liu, Huan

A2 - Pu, Calton

A2 - Hu, Xiaohua

A2 - Ahmed, Nesreen

A2 - Qiao, Mu

A2 - Song, Yang

A2 - Kossmann, Donald

A2 - Liu, Bing

A2 - Lee, Kisung

A2 - Tang, Jiliang

A2 - He, Jingrui

A2 - Saltz, Jeffrey

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 10 December 2018 through 13 December 2018

ER -

Chen J, Guan Q, Liang X, Bryant P, Grubel P, McPherson A et al. Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere. In Abe N, Liu H, Pu C, Hu X, Ahmed N, Qiao M, Song Y, Kossmann D, Liu B, Lee K, Tang J, He J, Saltz J, editors, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 1737-1746. 8622572. (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). doi: 10.1109/BigData.2018.8622572

Build and Execution Environment (BEE): An Encapsulated Environment Enabling HPC Applications Running Everywhere

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this