A vision for managing extreme-scale data hoards

Jeremy Logan; Kshitij Mehta; Gerd Heber; Scott Klasky; Tahsin Kurc; Norbert Podhorszki; Patrick Widener; Matthew Wolf

doi:10.1109/ICDCS.2019.00179

A vision for managing extreme-scale data hoards

Jeremy Logan, Kshitij Mehta, Gerd Heber, Scott Klasky, Tahsin Kurc, Norbert Podhorszki, Patrick Widener, Matthew Wolf

Workflow Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.

Original language	English
Title of host publication	Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1806-1817
Number of pages	12
ISBN (Electronic)	9781728125190
DOIs	https://doi.org/10.1109/ICDCS.2019.00179
State	Published - Jul 2019
Event	39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019 - Richardson, United States Duration: Jul 7 2019 → Jul 9 2019

Publication series

Name	Proceedings - International Conference on Distributed Computing Systems
Volume	2019-July

Conference

Conference	39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019
Country/Territory	United States
City	Richardson
Period	07/7/19 → 07/9/19

Funding

Without the continued support from the Department of Energy’s Office of Advanced Scientific Computing Research, the projects upon which this future vision rests, including SIRIUS, MONA, and SENSEI, would not be possible. Additionally, support from the DOE computing facilities in Oak Ridge and NERSC, as well as the National Science Foundation, was also critical. This work was also supported in part by 1U24CA180924-01A1, 3U24CA215109-02, and 1UG3CA225021-01 from the National Cancer Institute, R01LM011119-01 and R01LM009239 from the U.S. National Library of Medicine.

Funders	Funder number
SENSEI
National Science Foundation
U.S. Department of Energy
National Cancer Institute	R01LM009239, R01LM011119-01
U.S. National Library of Medicine
Advanced Scientific Computing Research
Oak Ridge Associated Universities

Keywords

Data provenance
Metadata management
Reproducibility
Scientific data management

Access to Document

10.1109/ICDCS.2019.00179

Cite this

Logan, J., Mehta, K., Heber, G., Klasky, S., Kurc, T., Podhorszki, N., Widener, P., & Wolf, M. (2019). A vision for managing extreme-scale data hoards. In Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019 (pp. 1806-1817). Article 8885312 (Proceedings - International Conference on Distributed Computing Systems; Vol. 2019-July). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICDCS.2019.00179

@inproceedings{aba1bd077f394ff18e9210065ca59494,

title = "A vision for managing extreme-scale data hoards",

abstract = "Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.",

keywords = "Data provenance, Metadata management, Reproducibility, Scientific data management",

author = "Jeremy Logan and Kshitij Mehta and Gerd Heber and Scott Klasky and Tahsin Kurc and Norbert Podhorszki and Patrick Widener and Matthew Wolf",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019 ; Conference date: 07-07-2019 Through 09-07-2019",

year = "2019",

month = jul,

doi = "10.1109/ICDCS.2019.00179",

language = "English",

series = "Proceedings - International Conference on Distributed Computing Systems",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1806--1817",

booktitle = "Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019",

}

Logan, J , Mehta, K, Heber, G, Klasky, S, Kurc, T, Podhorszki, N , Widener, P & Wolf, M 2019, A vision for managing extreme-scale data hoards. in Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019., 8885312, Proceedings - International Conference on Distributed Computing Systems, vol. 2019-July, Institute of Electrical and Electronics Engineers Inc., pp. 1806-1817, 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019, Richardson, United States, 07/7/19. https://doi.org/10.1109/ICDCS.2019.00179

A vision for managing extreme-scale data hoards. / Logan, Jeremy ; Mehta, Kshitij; Heber, Gerd et al.
Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 1806-1817 8885312 (Proceedings - International Conference on Distributed Computing Systems; Vol. 2019-July).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A vision for managing extreme-scale data hoards

AU - Logan, Jeremy

AU - Mehta, Kshitij

AU - Heber, Gerd

AU - Klasky, Scott

AU - Kurc, Tahsin

AU - Podhorszki, Norbert

AU - Widener, Patrick

AU - Wolf, Matthew

PY - 2019/7

Y1 - 2019/7

N2 - Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.

AB - Scientific data collections grow ever larger, both in terms of the size of individual data items and of the number and complexity of items. To use and manage them, it is important to directly address issues of robust and actionable provenance. We identify three key drivers as our focus: managing the size and complexity of metadata, lack of a priori information to match usage intents between publishers and consumers of data, and support for campaigns over collections of data driven by multi-disciplinary, collaborating teams. We introduce the Hoarde abstraction as an attempt to formalize a way of looking at collections of data to make them more tractable for later use. Hoarde leverages middleware and systems infrastructures for scientific and technical data management. Through the lens of a select group of challenging data usage scenarios, we discuss some of the aspects of implementation, usage, and forward portability of this new view on data management.

KW - Data provenance

KW - Metadata management

KW - Reproducibility

KW - Scientific data management

UR - http://www.scopus.com/inward/record.url?scp=85074827212&partnerID=8YFLogxK

U2 - 10.1109/ICDCS.2019.00179

DO - 10.1109/ICDCS.2019.00179

M3 - Conference contribution

AN - SCOPUS:85074827212

T3 - Proceedings - International Conference on Distributed Computing Systems

SP - 1806

EP - 1817

BT - Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019

Y2 - 7 July 2019 through 9 July 2019

ER -

Logan J , Mehta K, Heber G, Klasky S, Kurc T, Podhorszki N et al. A vision for managing extreme-scale data hoards. In Proceedings - 2019 39th IEEE International Conference on Distributed Computing Systems, ICDCS 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 1806-1817. 8885312. (Proceedings - International Conference on Distributed Computing Systems). doi: 10.1109/ICDCS.2019.00179

A vision for managing extreme-scale data hoards

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this