EMPRESS-Extensible metadata provider for extreme-scale scientific simulations

Margaret Lawson; Jay Lofstead; Scott Levy; Patrick Widener; Craig Ulmer; Shyamali Mukherjee; Gary Templet; Todd Kordenbrock

doi:10.1145/3149393.3149403

EMPRESS-Extensible metadata provider for extreme-scale scientific simulations

Margaret Lawson, Jay Lofstead, Scott Levy, Patrick Widener, Craig Ulmer, Shyamali Mukherjee, Gary Templet, Todd Kordenbrock

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrieval by offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with the storage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.

Original language	English
Title of host publication	Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017
Subtitle of host publication	The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Association for Computing Machinery, Inc
Pages	19-24
Number of pages	6
ISBN (Print)	9781450351348
DOIs	https://doi.org/10.1145/3149393.3149403
State	Published - Nov 12 2017
Externally published	Yes
Event	2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2017 - Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 - Denver, United States Duration: Nov 13 2017 → …

Publication series

Name	Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2017 - Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017
Country/Territory	United States
City	Denver
Period	11/13/17 → …

Keywords

ATDM
Custom metadata
Data tagging
Decaf
Metadata
SIRIUS

Access to Document

10.1145/3149393.3149403

Cite this

Lawson, M., Lofstead, J., Levy, S., Widener, P., Ulmer, C., Mukherjee, S., Templet, G., & Kordenbrock, T. (2017). EMPRESS-Extensible metadata provider for extreme-scale scientific simulations. In Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 19-24). (Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis). Association for Computing Machinery, Inc. https://doi.org/10.1145/3149393.3149403

Lawson, Margaret ; Lofstead, Jay ; Levy, Scott et al. / EMPRESS-Extensible metadata provider for extreme-scale scientific simulations. Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2017. pp. 19-24 (Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{9af50568772740769d677d23966869cd,

title = "EMPRESS-Extensible metadata provider for extreme-scale scientific simulations",

abstract = "Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrieval by offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with the storage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.",

keywords = "ATDM, Custom metadata, Data tagging, Decaf, Metadata, SIRIUS",

author = "Margaret Lawson and Jay Lofstead and Scott Levy and Patrick Widener and Craig Ulmer and Shyamali Mukherjee and Gary Templet and Todd Kordenbrock",

note = "Publisher Copyright: {\textcopyright} 2017 ACM.; 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2017 - Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017 ; Conference date: 13-11-2017",

year = "2017",

month = nov,

day = "12",

doi = "10.1145/3149393.3149403",

language = "English",

isbn = "9781450351348",

series = "Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Association for Computing Machinery, Inc",

pages = "19--24",

booktitle = "Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017",

}

Lawson, M, Lofstead, J, Levy, S, Widener, P, Ulmer, C, Mukherjee, S, Templet, G & Kordenbrock, T 2017, EMPRESS-Extensible metadata provider for extreme-scale scientific simulations. in Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis. Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis, Association for Computing Machinery, Inc, pp. 19-24, 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2017 - Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017, Denver, United States, 11/13/17. https://doi.org/10.1145/3149393.3149403

EMPRESS-Extensible metadata provider for extreme-scale scientific simulations. / Lawson, Margaret; Lofstead, Jay; Levy, Scott et al.
Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc, 2017. p. 19-24 (Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - EMPRESS-Extensible metadata provider for extreme-scale scientific simulations

AU - Lawson, Margaret

AU - Lofstead, Jay

AU - Levy, Scott

AU - Widener, Patrick

AU - Ulmer, Craig

AU - Mukherjee, Shyamali

AU - Templet, Gary

AU - Kordenbrock, Todd

PY - 2017/11/12

Y1 - 2017/11/12

N2 - Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrieval by offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with the storage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.

AB - Significant challenges exist in the efficient retrieval of data from extreme-scale simulations. An important and evolving method of addressing these challenges is application-level metadata management. Historically, HDF5 and NetCDF have eased data retrieval by offering rudimentary attribute capabilities that provide basic metadata. ADIOS simplified data retrieval by utilizing metadata for each process' data. EMPRESS provides a simple example of the next step in this evolution by integrating per-process metadata with the storage system itself, making it more broadly useful than single file or application formats. Additionally, it allows for more robust and customizable metadata.

KW - ATDM

KW - Custom metadata

KW - Data tagging

KW - Decaf

KW - Metadata

KW - SIRIUS

UR - http://www.scopus.com/inward/record.url?scp=85045872946&partnerID=8YFLogxK

U2 - 10.1145/3149393.3149403

DO - 10.1145/3149393.3149403

M3 - Conference contribution

AN - SCOPUS:85045872946

SN - 9781450351348

T3 - Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 19

EP - 24

BT - Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017

PB - Association for Computing Machinery, Inc

T2 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems, PDSW-DISCS 2017 - Held in conjunction with the International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017

Y2 - 13 November 2017

ER -

Lawson M, Lofstead J, Levy S, Widener P, Ulmer C, Mukherjee S et al. EMPRESS-Extensible metadata provider for extreme-scale scientific simulations. In Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis. Association for Computing Machinery, Inc. 2017. p. 19-24. (Proceedings of PDSW-DISCS 2017 - 2nd Joint International Workshop on Parallel Data Storage and Data Intensive Scalable Computing Systems - Held in conjunction with SC 2017: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1145/3149393.3149403

EMPRESS-Extensible metadata provider for extreme-scale scientific simulations

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this