The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale

E. Wes Bethel; Burlen Loring; Utkarsh Ayachit; David Camp; Earl P.N. Duque; Nicola Ferrier; Joseph Insley; Junmin Gu; James Kress; Patrick O’Leary; David Pugmire; Silvio Rizzi; David Thompson; Gunther H. Weber; Brad Whitlock; Matthew Wolf; Kesheng Wu

doi:10.1007/978-3-030-81627-8_13

The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale

E. Wes Bethel, Burlen Loring, Utkarsh Ayachit, David Camp, Earl P.N. Duque, Nicola Ferrier, Joseph Insley, Junmin Gu, James Kress, Patrick O’Leary, David Pugmire, Silvio Rizzi, David Thompson, Gunther H. Weber, Brad Whitlock, Matthew Wolf, Kesheng Wu

Visualization

Research output: Chapter in Book/Report/Conference proceeding › Chapter › peer-review

4 Scopus citations

Abstract

One key challenge when doing in situ processing is the investment required to add code to numerical simulations needed to take advantage of in situ processing. Such instrumentation code is often specialized, and tailored to a specific in situ method or infrastructure. Then, if a simulation wants to use other in situ tools, each of which has its own bespoke API [4], then the simulation code team will quickly become overwhelmed with having a different set of instrumentation APIs, one per in situ tool or method. In an ideal situation, such instrumentation need happen only once, and then the instrumentation API provides access to a large diversity of tools. In this way, a data producer’s instrumentation need not be modified if the user desires to take advantage of a different set of in situ tools. The SENSEI generic in situ interface addresses this challenge, which means that SENSEI-instrumented codes enjoy the benefit of being able to use a diversity of tools at scale, tools that include Libsim, Catalyst, Ascent, as well as user-defined methods written in C++ or Python. SENSEI has been shown to scale to greater than 1M-way concurrency on HPC platforms, and provides support for a rich and diverse collection of common scientific data models. This chapter presents the key design challenges that enable tool and processing portability at scale, some performance analysis, and example science applications of the methods.

Original language	English
Title of host publication	Mathematics and Visualization
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	281-306
Number of pages	26
DOIs	https://doi.org/10.1007/978-3-030-81627-8_13
State	Published - 2022

Publication series

Name	Mathematics and Visualization
ISSN (Print)	1612-3786
ISSN (Electronic)	2197-666X

Funding

Acknowledgements This work was supported by the Director, Office of Science, Office of Advanced Scientific Computing Research, of the U.S. Department of Energy under Contract Nos. DE-AC02-05CH11231 and DE-AC01-06CH11357, through the grant “Scalable Analysis Methods and In Situ Infrastructure for Extreme Scale Knowledge Discovery,” program manager Dr. Laura Biven. This research used resources of the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC02-05CH11231. Argonne National Laboratory’s work was supported by and used the resources of the Argonne Leadership Computing Facility, which is a U.S. Department of Energy, Office of Science User Facility supported under contract DE-AC02-06CH11357.

Funders	Funder number
U.S. Department of Energy	DE-AC02-05CH11231, DE-AC01-06CH11357
Office of Science	DE-AC02-06CH11357
Advanced Scientific Computing Research

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1007/978-3-030-81627-8_13

Cite this

Bethel, E. W., Loring, B., Ayachit, U., Camp, D., Duque, E. P. N., Ferrier, N., Insley, J., Gu, J., Kress, J., O’Leary, P., Pugmire, D., Rizzi, S., Thompson, D., Weber, G. H., Whitlock, B., Wolf, M., & Wu, K. (2022). The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale. In Mathematics and Visualization (pp. 281-306). (Mathematics and Visualization). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-81627-8_13

@inbook{47b128383d304df0a90b87ad4ef7c764,

title = "The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale",

abstract = "One key challenge when doing in situ processing is the investment required to add code to numerical simulations needed to take advantage of in situ processing. Such instrumentation code is often specialized, and tailored to a specific in situ method or infrastructure. Then, if a simulation wants to use other in situ tools, each of which has its own bespoke API [4], then the simulation code team will quickly become overwhelmed with having a different set of instrumentation APIs, one per in situ tool or method. In an ideal situation, such instrumentation need happen only once, and then the instrumentation API provides access to a large diversity of tools. In this way, a data producer{\textquoteright}s instrumentation need not be modified if the user desires to take advantage of a different set of in situ tools. The SENSEI generic in situ interface addresses this challenge, which means that SENSEI-instrumented codes enjoy the benefit of being able to use a diversity of tools at scale, tools that include Libsim, Catalyst, Ascent, as well as user-defined methods written in C++ or Python. SENSEI has been shown to scale to greater than 1M-way concurrency on HPC platforms, and provides support for a rich and diverse collection of common scientific data models. This chapter presents the key design challenges that enable tool and processing portability at scale, some performance analysis, and example science applications of the methods.",

author = "Bethel, {E. Wes} and Burlen Loring and Utkarsh Ayachit and David Camp and Duque, {Earl P.N.} and Nicola Ferrier and Joseph Insley and Junmin Gu and James Kress and Patrick O{\textquoteright}Leary and David Pugmire and Silvio Rizzi and David Thompson and Weber, {Gunther H.} and Brad Whitlock and Matthew Wolf and Kesheng Wu",

note = "Publisher Copyright: {\textcopyright} 2022, The Author(s), under exclusive license to Springer Nature Switzerland AG.",

year = "2022",

doi = "10.1007/978-3-030-81627-8_13",

language = "English",

series = "Mathematics and Visualization",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "281--306",

booktitle = "Mathematics and Visualization",

}

Bethel, EW, Loring, B, Ayachit, U, Camp, D, Duque, EPN, Ferrier, N, Insley, J, Gu, J, Kress, J, O’Leary, P, Pugmire, D, Rizzi, S, Thompson, D, Weber, GH, Whitlock, B, Wolf, M & Wu, K 2022, The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale. in Mathematics and Visualization. Mathematics and Visualization, Springer Science and Business Media Deutschland GmbH, pp. 281-306. https://doi.org/10.1007/978-3-030-81627-8_13

TY - CHAP

T1 - The SENSEI Generic In Situ Interface

T2 - Tool and Processing Portability at Scale

AU - Bethel, E. Wes

AU - Loring, Burlen

AU - Ayachit, Utkarsh

AU - Camp, David

AU - Duque, Earl P.N.

AU - Ferrier, Nicola

AU - Insley, Joseph

AU - Gu, Junmin

AU - Kress, James

AU - O’Leary, Patrick

AU - Pugmire, David

AU - Rizzi, Silvio

AU - Thompson, David

AU - Weber, Gunther H.

AU - Whitlock, Brad

AU - Wolf, Matthew

AU - Wu, Kesheng

PY - 2022

Y1 - 2022

N2 - One key challenge when doing in situ processing is the investment required to add code to numerical simulations needed to take advantage of in situ processing. Such instrumentation code is often specialized, and tailored to a specific in situ method or infrastructure. Then, if a simulation wants to use other in situ tools, each of which has its own bespoke API [4], then the simulation code team will quickly become overwhelmed with having a different set of instrumentation APIs, one per in situ tool or method. In an ideal situation, such instrumentation need happen only once, and then the instrumentation API provides access to a large diversity of tools. In this way, a data producer’s instrumentation need not be modified if the user desires to take advantage of a different set of in situ tools. The SENSEI generic in situ interface addresses this challenge, which means that SENSEI-instrumented codes enjoy the benefit of being able to use a diversity of tools at scale, tools that include Libsim, Catalyst, Ascent, as well as user-defined methods written in C++ or Python. SENSEI has been shown to scale to greater than 1M-way concurrency on HPC platforms, and provides support for a rich and diverse collection of common scientific data models. This chapter presents the key design challenges that enable tool and processing portability at scale, some performance analysis, and example science applications of the methods.

AB - One key challenge when doing in situ processing is the investment required to add code to numerical simulations needed to take advantage of in situ processing. Such instrumentation code is often specialized, and tailored to a specific in situ method or infrastructure. Then, if a simulation wants to use other in situ tools, each of which has its own bespoke API [4], then the simulation code team will quickly become overwhelmed with having a different set of instrumentation APIs, one per in situ tool or method. In an ideal situation, such instrumentation need happen only once, and then the instrumentation API provides access to a large diversity of tools. In this way, a data producer’s instrumentation need not be modified if the user desires to take advantage of a different set of in situ tools. The SENSEI generic in situ interface addresses this challenge, which means that SENSEI-instrumented codes enjoy the benefit of being able to use a diversity of tools at scale, tools that include Libsim, Catalyst, Ascent, as well as user-defined methods written in C++ or Python. SENSEI has been shown to scale to greater than 1M-way concurrency on HPC platforms, and provides support for a rich and diverse collection of common scientific data models. This chapter presents the key design challenges that enable tool and processing portability at scale, some performance analysis, and example science applications of the methods.

UR - http://www.scopus.com/inward/record.url?scp=85130092876&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-81627-8_13

DO - 10.1007/978-3-030-81627-8_13

M3 - Chapter

AN - SCOPUS:85130092876

T3 - Mathematics and Visualization

SP - 281

EP - 306

BT - Mathematics and Visualization

PB - Springer Science and Business Media Deutschland GmbH

ER -

The SENSEI Generic In Situ Interface: Tool and Processing Portability at Scale

Abstract

Publication series

Funding

UN SDGs

Access to Document

Other files and links

Fingerprint

Cite this