Experimental evaluation of infiniband transport over local- and wide-area networks

Steven Carter; Makia Minich; Nageswara S.V. Rao

Experimental evaluation of infiniband transport over local- and wide-area networks

Steven Carter, Makia Minich, Nageswara S.V. Rao

Autonomous and Complex Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.

Original language	English
Title of host publication	Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007
Publisher	Association for Computing Machinery, Inc
Pages	419-426
Number of pages	8
ISBN (Electronic)	1565553136, 9781565553132
State	Published - Mar 25 2007
Event	2007 Agent Directed Simulation Symposium, ADS 2007 - Norfolk, United States Duration: Mar 25 2007 → Mar 29 2007

Publication series

Name	Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007

Conference

Conference	2007 Agent Directed Simulation Symposium, ADS 2007
Country/Territory	United States
City	Norfolk
Period	03/25/07 → 03/29/07

Funding

This research is sponsored by the High Performance Networking Program of the Office of Science, U.S. Department of Energy, under Contract No. DE-AC05-00OR22725 with UT-Battelle, LLC, and by National Science Foundation Under Grant No. ANI-0335185.

Keywords

Infiniband
Lustre file system
Storage system
Supercomputers
Wide-area connections

Cite this

Carter, S., Minich, M., & Rao, N. S. V. (2007). Experimental evaluation of infiniband transport over local- and wide-area networks. In Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007 (pp. 419-426). (Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007). Association for Computing Machinery, Inc.

Carter, Steven ; Minich, Makia ; Rao, Nageswara S.V. / Experimental evaluation of infiniband transport over local- and wide-area networks. Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007. Association for Computing Machinery, Inc, 2007. pp. 419-426 (Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007).

@inproceedings{c46630a7388742d1b4896ed2292002ef,

title = "Experimental evaluation of infiniband transport over local- and wide-area networks",

abstract = "High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.",

keywords = "Infiniband, Lustre file system, Storage system, Supercomputers, Wide-area connections",

author = "Steven Carter and Makia Minich and Rao, {Nageswara S.V.}",

year = "2007",

month = mar,

day = "25",

language = "English",

series = "Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007",

publisher = "Association for Computing Machinery, Inc",

pages = "419--426",

booktitle = "Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007",

note = "2007 Agent Directed Simulation Symposium, ADS 2007 ; Conference date: 25-03-2007 Through 29-03-2007",

}

Carter, S, Minich, M & Rao, NSV 2007, Experimental evaluation of infiniband transport over local- and wide-area networks. in Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007. Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007, Association for Computing Machinery, Inc, pp. 419-426, 2007 Agent Directed Simulation Symposium, ADS 2007, Norfolk, United States, 03/25/07.

Experimental evaluation of infiniband transport over local- and wide-area networks. / Carter, Steven; Minich, Makia; Rao, Nageswara S.V.
Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007. Association for Computing Machinery, Inc, 2007. p. 419-426 (Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Experimental evaluation of infiniband transport over local- and wide-area networks

AU - Carter, Steven

AU - Minich, Makia

AU - Rao, Nageswara S.V.

PY - 2007/3/25

Y1 - 2007/3/25

N2 - High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.

AB - High-performance file systems that span across supercomputers and storage systems are becoming increasingly important, as these resources are being dispersed both within the enterprise and across the wide-area. Infiniband transport has been proposed as a candidate to provide high-performance access over interconnects primarily within an enterprise. In this paper, we present an experimental study of Infiniband transport over ORNL infrastructure of Cray XT3 supercomputer and cluster storage system for local-area and DOE UltraScience Network (USN) for wide-area. Our results indicate that Infiniband throughput of 7.6Gbps (4x) scales to 8,000 miles with 5% degradation with no customization over 10Gbps wide-area connections. We conducted preliminary performance tests of Lustre file system over these connections, which identified potential parameter and modules optimizations.

KW - Infiniband

KW - Lustre file system

KW - Storage system

KW - Supercomputers

KW - Wide-area connections

UR - http://www.scopus.com/inward/record.url?scp=83055199722&partnerID=8YFLogxK

M3 - Conference contribution

AN - SCOPUS:83055199722

T3 - Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007

SP - 419

EP - 426

BT - Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007

PB - Association for Computing Machinery, Inc

T2 - 2007 Agent Directed Simulation Symposium, ADS 2007

Y2 - 25 March 2007 through 29 March 2007

ER -

Carter S, Minich M, Rao NSV. Experimental evaluation of infiniband transport over local- and wide-area networks. In Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007. Association for Computing Machinery, Inc. 2007. p. 419-426. (Agent Directed Simulation Symposium, ADS 2007 - Proceedings of the 2007 Spring Simulation Multiconference, SpringSim 2007).

Experimental evaluation of infiniband transport over local- and wide-area networks

Abstract

Publication series

Conference

Funding

Keywords

Other files and links

Fingerprint

Cite this