LADS: Optimizing data transfers using layout-aware data scheduling

Youngjae Kim, Scott Atchley, Geoffroy R. Vallée, Galen M. Shipman

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

32 Scopus citations

Abstract

While future terabit networks hold the promise of significantly improving big-data motion among geographically distributed data centers, significant challenges must be overcome even on today’s 100 gigabit networks to realize end-to-end performance. Multiple bottlenecks exist along the end-to-end path from source to sink. Data storage infrastructure at both the source and sink and its interplay with the wide-area network are increasingly the bottleneck to achieving high performance. In this paper, we identify the issues that lead to congestion on the path of an end-to-end data transfer in the terabit network environment, and we present a new bulk data movement framework called LADS for terabit networks. LADS exploits the underlying storage layout at each endpoint to maximize throughput without negatively impacting the performance of shared storage resources for other users. LADS also uses the Common Communication Interface (CCI) in lieu of the sockets interface to use zero-copy, OS-bypass hardware when available. It can further improve data transfer performance under congestion on the end systems using buffering at the source using flash storage. With our evaluations, we show that LADS can avoid congested storage elements within the shared storage resource, improving I/O bandwidth, and data transfer rates across the high speed networks.

Original languageEnglish
Title of host publicationProceedings of the 13th USENIX Conference on File and Storage Technologies, FAST 2015
PublisherUSENIX Association
Pages67-80
Number of pages14
ISBN (Electronic)9781931971201
StatePublished - 2015
Event13th USENIX Conference on File and Storage Technologies, FAST 2015 - Santa Clara, United States
Duration: Feb 16 2015Feb 19 2015

Publication series

NameProceedings of the 13th USENIX Conference on File and Storage Technologies, FAST 2015

Conference

Conference13th USENIX Conference on File and Storage Technologies, FAST 2015
Country/TerritoryUnited States
CitySanta Clara
Period02/16/1502/19/15

Funding

We thank the reviewers and our shepherd, Nitin Agrawal, for their constructive comments that have significantly improved the paper. This research is sponsored by the Office of Advanced Scientific Computing Research, U.S. Department of Energy and used resources of the Oak Ridge Leadership Computing Facility, located in the National Center for Computational Sciences at Oak Ridge National Laboratory, which is supported by the Office of Science of the Department of Energy under Contract DE-AC05-00OR22725.

Fingerprint

Dive into the research topics of 'LADS: Optimizing data transfers using layout-aware data scheduling'. Together they form a unique fingerprint.

Cite this