TY - GEN
T1 - TagIt
T2 - 2017 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2017
AU - Sim, Hyogi
AU - Kim, Youngjae
AU - Vazhkudai, Sudharshan S.
AU - Vallee, Geoffroy R.
AU - Lim, Seung Hwan
AU - Butt, Ali R.
N1 - Publisher Copyright:
© 2017 ACM.
PY - 2017
Y1 - 2017
N2 - Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capability to support data discovery. The tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. Our evaluation shows that TagIt can expedite data search by up to 10 × over the extant decoupled approach.
AB - Data services such as search, discovery, and management in scalable distributed environments have traditionally been decoupled from the underlying file systems, and are often deployed using external databases and indexing services. However, modern data production rates, looming data movement costs, and the lack of metadata, entail revisiting the decoupled file system-data services design philosophy. In this paper, we present TagIt, a scalable data management service framework aimed at scientific datasets, which is tightly integrated into a shared-nothing distributed file system. A key feature of TagIt is a scalable, distributed metadata indexing framework, using which we implement a flexible tagging capability to support data discovery. The tags can also be associated with an active operator, for pre-processing, filtering, or automatic metadata extraction, which we seamlessly offload to file servers in a load-aware fashion. Our evaluation shows that TagIt can expedite data search by up to 10 × over the extant decoupled approach.
KW - Distributed file systems
KW - Indexing methods
KW - Search process
UR - http://www.scopus.com/inward/record.url?scp=85142291500&partnerID=8YFLogxK
U2 - 10.1145/3126908.3126929
DO - 10.1145/3126908.3126929
M3 - Conference contribution
AN - SCOPUS:85142291500
T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC
BT - SC 2017 - International Conference for High Performance Computing, Networking, Storage and Analysis
PB - IEEE Computer Society
Y2 - 12 November 2017 through 17 November 2017
ER -