Wide-area lustre file system using LNet routers

Nageswara S.V. Rao; Neena Imam; Jessey Hanley; Sarp Oral

doi:10.1109/SYSCON.2018.8369541

Wide-area lustre file system using LNet routers

Nageswara S.V. Rao, Neena Imam, Jessey Hanley, Sarp Oral

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

6 Scopus citations

Abstract

Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

Original language	English
Title of host publication	12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-6
Number of pages	6
ISBN (Electronic)	9781538636640
DOIs	https://doi.org/10.1109/SYSCON.2018.8369541
State	Published - May 30 2018
Event	12th Annual IEEE International Systems Conference, SysCon 2018 - Vancouver, Canada Duration: Apr 24 2018 → Apr 26 2018

Publication series

Name	12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings

Conference

Conference	12th Annual IEEE International Systems Conference, SysCon 2018
Country/Territory	Canada
City	Vancouver
Period	04/24/18 → 04/26/18

Funding

This work is funded by the Mathematics of Complex, Distributed, Interconnected Systems Program, Office of Advanced Computing Research, U.S. Department of Energy, and by Extreme Scale Systems Center, sponsored by U. S. Department of Defense, and performed at Oak Ridge National Laboratory managed by UT-Battelle, LLC for U.S. Department of Energy under Contract No. DE-AC05-00OR22725. This work is funded by the Mathematics of Complex, Distributed, Interconnected Systems Program, Office of Advanced Computing Research, U.S. Department of Energy, and by Extreme Scale Systems Center, sponsored by U. S. Department of Defense, and performed at Oak Ridge National Laboratory managed by UT-Battelle, LLC for U.S. Department of Energy under Contract No. DE-AC05-00OR22725

Keywords

Lustre
Network measurements
Throughput
Wide-area networks

Access to Document

10.1109/SYSCON.2018.8369541

Cite this

Rao, N. S. V., Imam, N., Hanley, J., & Oral, S. (2018). Wide-area lustre file system using LNet routers. In 12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings (pp. 1-6). (12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/SYSCON.2018.8369541

@inproceedings{cbdb2ec4a28a4b1599ffc658a684d631,

title = "Wide-area lustre file system using LNet routers",

abstract = "Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.",

keywords = "Lustre, Network measurements, Throughput, Wide-area networks",

author = "Rao, {Nageswara S.V.} and Neena Imam and Jessey Hanley and Sarp Oral",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 12th Annual IEEE International Systems Conference, SysCon 2018 ; Conference date: 24-04-2018 Through 26-04-2018",

year = "2018",

month = may,

day = "30",

doi = "10.1109/SYSCON.2018.8369541",

language = "English",

series = "12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--6",

booktitle = "12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings",

}

Rao, NSV, Imam, N, Hanley, J & Oral, S 2018, Wide-area lustre file system using LNet routers. in 12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings. 12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings, Institute of Electrical and Electronics Engineers Inc., pp. 1-6, 12th Annual IEEE International Systems Conference, SysCon 2018, Vancouver, Canada, 04/24/18. https://doi.org/10.1109/SYSCON.2018.8369541

Wide-area lustre file system using LNet routers. / Rao, Nageswara S.V.; Imam, Neena; Hanley, Jessey et al.
12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings. Institute of Electrical and Electronics Engineers Inc., 2018. p. 1-6 (12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Wide-area lustre file system using LNet routers

AU - Rao, Nageswara S.V.

AU - Imam, Neena

AU - Hanley, Jessey

AU - Oral, Sarp

PY - 2018/5/30

Y1 - 2018/5/30

N2 - Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

AB - Scientific and big data computations are increasingly being distributed across wide-area networks, and they often require access to remote files. The file systems that are directly mounted over wide-area networks transparently support such computations, and also obviate the need for special purpose file transfer tools. In typical distributed file systems, the access is limited to local sites, and in particular, the reach of Lustre file system implemented over InfiniBand (IB) is limited to at most tens of miles due to 2.5ms latency bound. We describe LNet router methods that connect IB Lustre file system to remote Ethernet clients over wide-area networks. We collect extensive Lustre throughput measurements over 10Gbps connections with 0-366ms round-trip times. They demonstrate that Gbps throughput can be sustained over connections spanning the globe. We present Lustre throughput profiles over local and wide-area connections, which show the effects of various buffers and credits; in particular, they highlight the throughput limits for large transfers over wide-area connections. Furthermore, the measurements show the positive effects of pipelining in achieving higher throughput for successively file transfers compared to rates indicated by IOzone benchmark rates.

KW - Lustre

KW - Network measurements

KW - Throughput

KW - Wide-area networks

UR - http://www.scopus.com/inward/record.url?scp=85048878515&partnerID=8YFLogxK

U2 - 10.1109/SYSCON.2018.8369541

DO - 10.1109/SYSCON.2018.8369541

M3 - Conference contribution

AN - SCOPUS:85048878515

T3 - 12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings

SP - 1

EP - 6

BT - 12th Annual IEEE International Systems Conference, SysCon 2018 - Proceedings

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 12th Annual IEEE International Systems Conference, SysCon 2018

Y2 - 24 April 2018 through 26 April 2018

ER -

Wide-area lustre file system using LNet routers

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this