PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs

Mihael Hategan-Marandiuc; Andre Merzky; Nicholson Collier; Ketan Maheshwari; Jonathan Ozik; Matteo Turilli; Andreas Wilke; Justin M. Wozniak; Kyle Chard; Ian Foster; Rafael Ferreira Da Silva; Shantenu Jha; Daniel Laney

doi:10.1109/e-Science58273.2023.10254912

PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs

Mihael Hategan-Marandiuc, Andre Merzky, Nicholson Collier, Ketan Maheshwari, Jonathan Ozik, Matteo Turilli, Andreas Wilke, Justin M. Wozniak, Kyle Chard, Ian Foster, Rafael Ferreira Da Silva, Shantenu Jha, Daniel Laney

Workflow and Ecosystem Services

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective use of allocations, and to co-locate compute jobs with large datasets. Unfortunately, moving scientific applications between HPC systems is challenging for various reasons, most notably that HPC systems have different HPC schedulers. We introduce PSI/J, a job management abstraction API intended to simplify the construction of software components and applications that are portable over various HPC scheduler implementations. We argue that such a system is both necessary and that no viable alternative currently exists. We analyze similar notable APIs and attempt to determine the factors that influenced their evolution and adoption by the HPC community. We base the design of PSI/J on that analysis. We describe how PSI/J has been integrated in three workflow systems and one application, and also show via experiments that PSI/J imposes minimal overhead.

Original language	English
Title of host publication	Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023
Publisher	Institute of Electrical and Electronics Engineers Inc.
ISBN (Electronic)	9798350322231
DOIs	https://doi.org/10.1109/e-Science58273.2023.10254912
State	Published - 2023
Event	19th IEEE International Conference on e-Science, e-Science 2023 - Limassol, Cyprus Duration: Oct 9 2023 → Oct 14 2023

Publication series

Name	Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023

Conference

Conference	19th IEEE International Conference on e-Science, e-Science 2023
Country/Territory	Cyprus
City	Limassol
Period	10/9/23 → 10/14/23

Funding

supported by the Office of Science of the U.S. DOE under Contract No. DE-AC05-00OR22725. OSPREY work was supported by the National Science Foundation under Grant No. 2200234, the National Institutes of Health under grant R01DA055502, and the DOE Office of Science through the Bio-preparedness Research Virtual Environment (BRaVE) initiative. This manuscript has been authored in part by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The publisher, by accepting the article for publication, acknowledges that the U.S. Government retains a non-exclusive, paid up, irrevocable, worldwide license to publish or reproduce the published form of the manuscript, or allow others to do so, for U.S. Government purposes. The DOE will provide public access to these results in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). ACKNOWLEDGEMENTS This research was supported by the Exascale Computing Project (17-SC-20-SC), a collaborative effort of the U.S. Department of Energy Office of Science and the National Nuclear Security Administration. This work was performed under the auspices of the U.S. Department of Energy by Lawrence Livermore National Laboratory under Contract DE-AC52-07NA27344 (LLNL-CONF-826133), Argonne National Laboratory under Contract DE-AC02-06CH11357, and Brookhaven National Laboratory under Contract DESC0012704. This research used resources of the OLCF at ORNL, which is

Funders	Funder number
National Science Foundation	2200234
National Institutes of Health	R01DA055502
U.S. Department of Energy	DE-AC05-00OR22725
Office of Science
National Nuclear Security Administration
Argonne National Laboratory	DE-AC02-06CH11357
Lawrence Livermore National Laboratory	LLNL-CONF-826133, DE-AC52-07NA27344
Brookhaven National Laboratory	DESC0012704

Access to Document

10.1109/e-Science58273.2023.10254912

Cite this

Hategan-Marandiuc, M., Merzky, A., Collier, N., Maheshwari, K., Ozik, J., Turilli, M., Wilke, A., Wozniak, J. M., Chard, K., Foster, I., Da Silva, R. F., Jha, S., & Laney, D. (2023). PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs. In Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023 (Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/e-Science58273.2023.10254912

Hategan-Marandiuc, Mihael ; Merzky, Andre ; Collier, Nicholson et al. / PSI/J : A Portable Interface for Submitting, Monitoring, and Managing Jobs. Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023).

@inproceedings{0b223c161e4b4511a91d1f4c3754eb31,

title = "PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs",

abstract = "It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective use of allocations, and to co-locate compute jobs with large datasets. Unfortunately, moving scientific applications between HPC systems is challenging for various reasons, most notably that HPC systems have different HPC schedulers. We introduce PSI/J, a job management abstraction API intended to simplify the construction of software components and applications that are portable over various HPC scheduler implementations. We argue that such a system is both necessary and that no viable alternative currently exists. We analyze similar notable APIs and attempt to determine the factors that influenced their evolution and adoption by the HPC community. We base the design of PSI/J on that analysis. We describe how PSI/J has been integrated in three workflow systems and one application, and also show via experiments that PSI/J imposes minimal overhead.",

author = "Mihael Hategan-Marandiuc and Andre Merzky and Nicholson Collier and Ketan Maheshwari and Jonathan Ozik and Matteo Turilli and Andreas Wilke and Wozniak, {Justin M.} and Kyle Chard and Ian Foster and {Da Silva}, {Rafael Ferreira} and Shantenu Jha and Daniel Laney",

note = "Publisher Copyright: {\textcopyright} 2023 IEEE.; 19th IEEE International Conference on e-Science, e-Science 2023 ; Conference date: 09-10-2023 Through 14-10-2023",

year = "2023",

doi = "10.1109/e-Science58273.2023.10254912",

language = "English",

series = "Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023",

}

Hategan-Marandiuc, M, Merzky, A, Collier, N, Maheshwari, K, Ozik, J, Turilli, M, Wilke, A, Wozniak, JM, Chard, K, Foster, I, Da Silva, RF, Jha, S & Laney, D 2023, PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs. in Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023. Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023, Institute of Electrical and Electronics Engineers Inc., 19th IEEE International Conference on e-Science, e-Science 2023, Limassol, Cyprus, 10/9/23. https://doi.org/10.1109/e-Science58273.2023.10254912

PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs. / Hategan-Marandiuc, Mihael; Merzky, Andre; Collier, Nicholson et al.
Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023. Institute of Electrical and Electronics Engineers Inc., 2023. (Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - PSI/J

T2 - 19th IEEE International Conference on e-Science, e-Science 2023

AU - Hategan-Marandiuc, Mihael

AU - Merzky, Andre

AU - Collier, Nicholson

AU - Maheshwari, Ketan

AU - Ozik, Jonathan

AU - Turilli, Matteo

AU - Wilke, Andreas

AU - Wozniak, Justin M.

AU - Chard, Kyle

AU - Foster, Ian

AU - Da Silva, Rafael Ferreira

AU - Jha, Shantenu

AU - Laney, Daniel

PY - 2023

Y1 - 2023

N2 - It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective use of allocations, and to co-locate compute jobs with large datasets. Unfortunately, moving scientific applications between HPC systems is challenging for various reasons, most notably that HPC systems have different HPC schedulers. We introduce PSI/J, a job management abstraction API intended to simplify the construction of software components and applications that are portable over various HPC scheduler implementations. We argue that such a system is both necessary and that no viable alternative currently exists. We analyze similar notable APIs and attempt to determine the factors that influenced their evolution and adoption by the HPC community. We base the design of PSI/J on that analysis. We describe how PSI/J has been integrated in three workflow systems and one application, and also show via experiments that PSI/J imposes minimal overhead.

AB - It is generally desirable for high-performance computing (HPC) applications to be portable between HPC systems, for example to make use of more performant hardware, make effective use of allocations, and to co-locate compute jobs with large datasets. Unfortunately, moving scientific applications between HPC systems is challenging for various reasons, most notably that HPC systems have different HPC schedulers. We introduce PSI/J, a job management abstraction API intended to simplify the construction of software components and applications that are portable over various HPC scheduler implementations. We argue that such a system is both necessary and that no viable alternative currently exists. We analyze similar notable APIs and attempt to determine the factors that influenced their evolution and adoption by the HPC community. We base the design of PSI/J on that analysis. We describe how PSI/J has been integrated in three workflow systems and one application, and also show via experiments that PSI/J imposes minimal overhead.

UR - http://www.scopus.com/inward/record.url?scp=85174248611&partnerID=8YFLogxK

U2 - 10.1109/e-Science58273.2023.10254912

DO - 10.1109/e-Science58273.2023.10254912

M3 - Conference contribution

AN - SCOPUS:85174248611

T3 - Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023

BT - Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 9 October 2023 through 14 October 2023

ER -

Hategan-Marandiuc M, Merzky A, Collier N, Maheshwari K, Ozik J, Turilli M et al. PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs. In Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023. Institute of Electrical and Electronics Engineers Inc. 2023. (Proceedings 2023 IEEE 19th International Conference on e-Science, e-Science 2023). doi: 10.1109/e-Science58273.2023.10254912

PSI/J: A Portable Interface for Submitting, Monitoring, and Managing Jobs

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this