Dredging a data lake: Decentralized metadata extraction

Tyler J. Skluzacek

doi:10.1145/3366624.3368170

Dredging a data lake: Decentralized metadata extraction

Tyler J. Skluzacek

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

7 Scopus citations

Abstract

The rapid generation of data from distributed IoT devices, scientific instruments, and compute clusters presents unique data management challenges. The influx of large, heterogeneous, and complex data causes repositories to become siloed or generally unsearchable-both problems not currently well-addressed by distributed file systems. In this work, we propose Xtract, a serverless middleware to extract metadata from files spread across heterogeneous edge computing resources. In my future work, we intend to study how Xtract can automatically construct file extraction workflows subject to users' cost, time, security, and compute allocation constraints. To this end, Xtract will enable the creation of a searchable centralized index across distributed data collections.

Original language	English
Title of host publication	Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019
Publisher	Association for Computing Machinery, Inc
Pages	51-53
Number of pages	3
ISBN (Electronic)	9781450370394
DOIs	https://doi.org/10.1145/3366624.3368170
State	Published - Dec 9 2019
Externally published	Yes
Event	20th International Middleware Conference Doctoral Symposium, Middleware 2019, Part of Middleware 2019 - Davis, United States Duration: Dec 9 2019 → Dec 13 2019

Publication series

Name	Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019

Conference

Conference	20th International Middleware Conference Doctoral Symposium, Middleware 2019, Part of Middleware 2019
Country/Territory	United States
City	Davis
Period	12/9/19 → 12/13/19

Funding

This research is conducted under the guidance of Dr. Ian Foster and Dr. Kyle Chard, and with contributions from Dr. Ryan Chard, Dr. Zhuozhao Li, Yadu Babuji, and Ryan Wong. We gratefully acknowledge the use of compute resources from the Jetstream cloud for science and engineering [12].

Keywords

Data lakes
File systems
Metadata extraction
Serverless

Access to Document

10.1145/3366624.3368170

Cite this

Skluzacek, T. J. (2019). Dredging a data lake: Decentralized metadata extraction. In Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019 (pp. 51-53). (Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3366624.3368170

Skluzacek, Tyler J. / Dredging a data lake : Decentralized metadata extraction. Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019. Association for Computing Machinery, Inc, 2019. pp. 51-53 (Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019).

@inproceedings{050033ea6cb445e59cd97b9ab0f4a1b7,

title = "Dredging a data lake: Decentralized metadata extraction",

abstract = "The rapid generation of data from distributed IoT devices, scientific instruments, and compute clusters presents unique data management challenges. The influx of large, heterogeneous, and complex data causes repositories to become siloed or generally unsearchable-both problems not currently well-addressed by distributed file systems. In this work, we propose Xtract, a serverless middleware to extract metadata from files spread across heterogeneous edge computing resources. In my future work, we intend to study how Xtract can automatically construct file extraction workflows subject to users' cost, time, security, and compute allocation constraints. To this end, Xtract will enable the creation of a searchable centralized index across distributed data collections.",

keywords = "Data lakes, File systems, Metadata extraction, Serverless",

author = "Skluzacek, \{Tyler J.\}",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computing Machinery.; 20th International Middleware Conference Doctoral Symposium, Middleware 2019, Part of Middleware 2019 ; Conference date: 09-12-2019 Through 13-12-2019",

year = "2019",

month = dec,

day = "9",

doi = "10.1145/3366624.3368170",

language = "English",

series = "Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019",

publisher = "Association for Computing Machinery, Inc",

pages = "51--53",

booktitle = "Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019",

}

Skluzacek, TJ 2019, Dredging a data lake: Decentralized metadata extraction. in Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019. Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019, Association for Computing Machinery, Inc, pp. 51-53, 20th International Middleware Conference Doctoral Symposium, Middleware 2019, Part of Middleware 2019, Davis, United States, 12/9/19. https://doi.org/10.1145/3366624.3368170

Dredging a data lake: Decentralized metadata extraction. / Skluzacek, Tyler J.
Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019. Association for Computing Machinery, Inc, 2019. p. 51-53 (Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Dredging a data lake

T2 - 20th International Middleware Conference Doctoral Symposium, Middleware 2019, Part of Middleware 2019

AU - Skluzacek, Tyler J.

PY - 2019/12/9

Y1 - 2019/12/9

N2 - The rapid generation of data from distributed IoT devices, scientific instruments, and compute clusters presents unique data management challenges. The influx of large, heterogeneous, and complex data causes repositories to become siloed or generally unsearchable-both problems not currently well-addressed by distributed file systems. In this work, we propose Xtract, a serverless middleware to extract metadata from files spread across heterogeneous edge computing resources. In my future work, we intend to study how Xtract can automatically construct file extraction workflows subject to users' cost, time, security, and compute allocation constraints. To this end, Xtract will enable the creation of a searchable centralized index across distributed data collections.

AB - The rapid generation of data from distributed IoT devices, scientific instruments, and compute clusters presents unique data management challenges. The influx of large, heterogeneous, and complex data causes repositories to become siloed or generally unsearchable-both problems not currently well-addressed by distributed file systems. In this work, we propose Xtract, a serverless middleware to extract metadata from files spread across heterogeneous edge computing resources. In my future work, we intend to study how Xtract can automatically construct file extraction workflows subject to users' cost, time, security, and compute allocation constraints. To this end, Xtract will enable the creation of a searchable centralized index across distributed data collections.

KW - Data lakes

KW - File systems

KW - Metadata extraction

KW - Serverless

UR - https://www.scopus.com/pages/publications/85078240684

U2 - 10.1145/3366624.3368170

DO - 10.1145/3366624.3368170

M3 - Conference contribution

AN - SCOPUS:85078240684

T3 - Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019

SP - 51

EP - 53

BT - Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019

PB - Association for Computing Machinery, Inc

Y2 - 9 December 2019 through 13 December 2019

ER -

Skluzacek TJ. Dredging a data lake: Decentralized metadata extraction. In Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019. Association for Computing Machinery, Inc. 2019. p. 51-53. (Middleware 2019 - Proceedings of the 2019 20th International Middleware Conference Doctoral Symposium, Part of Middleware 2019). doi: 10.1145/3366624.3368170

Dredging a data lake: Decentralized metadata extraction

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this