An evaluation of the CORAL interconnects

Christopher Zimmer; Scott Atchley; Ramesh Pankajakshan; Brian E. Smith; Ian Karlin; Matthew L. Leininger; Adam Bertsch; Brian S. Ryujin; Jason Burmark; André Walker-Loud; M. A. Clark; Olga Pearce

doi:10.1145/3295500.3356166

An evaluation of the CORAL interconnects

Christopher Zimmer, Scott Atchley, Ramesh Pankajakshan, Brian E. Smith, Ian Karlin, Matthew L. Leininger, Adam Bertsch, Brian S. Ryujin, Jason Burmark, André Walker-Loud, M. A. Clark, Olga Pearce

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

22 Scopus citations

Abstract

The US Department of Energy deployed the Summit and Sierra supercomputers with the latest state-of-the-art network interconnect technology in 2018 and both systems entered production in 2019. In this paper, we provide an in-depth assessment of the systems' network interconnects that are based on Enhanced Data Rate (EDR) 100 Gb/s Mellanox InfiniBand. Both systems use second-generation EDR Host Channel Adapters (HCAs) and switches with several new features such as Adaptive Routing (AR), switch-based collectives, and HCA-based tag matching. Although based on the same components, Summit's network is "non-blocking" (i.e., a fully provisioned Clos network) and Sierra's network has a 2:1 taper between the racks and aggregation switches. We evaluate the two systems' interconnects using traditional communication benchmarks as well as production applications. We find that the new Adaptive Routing dramatically improves performance but the other new features still need improvement.

Original language	English
Title of host publication	Proceedings of SC 2019
Subtitle of host publication	The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	IEEE Computer Society
ISBN (Electronic)	9781450362290
DOIs	https://doi.org/10.1145/3295500.3356166
State	Published - Nov 17 2019
Event	2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 - Denver, United States Duration: Nov 17 2019 → Nov 22 2019

Publication series

Name	International Conference for High Performance Computing, Networking, Storage and Analysis, SC
ISSN (Print)	2167-4329
ISSN (Electronic)	2167-4337

Conference

Conference	2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019
Country/Territory	United States
City	Denver
Period	11/17/19 → 11/22/19

Funding

This work was performed under the auspices of the U.S. DOE by Oak Ridge Leadership Computing Facility at ORNL under contract DE-AC05-00OR22725. The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Funders	Funder number
Oak Ridge National Laboratory	DE-AC05-00OR22725
U.S. Department of Energy

Keywords

Bandwidth
Congestion
EDR
High performance computing
InfiniBand
Interconnect
Latency
Offload
Switch collectives
Tag matching

Access to Document

10.1145/3295500.3356166

Cite this

Zimmer, C., Atchley, S., Pankajakshan, R., Smith, B. E., Karlin, I., Leininger, M. L., Bertsch, A., Ryujin, B. S., Burmark, J., Walker-Loud, A., Clark, M. A., & Pearce, O. (2019). An evaluation of the CORAL interconnects. In Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis Article a39 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). IEEE Computer Society. https://doi.org/10.1145/3295500.3356166

@inproceedings{ec1d58902e0945f3b1afbec707be2899,

title = "An evaluation of the CORAL interconnects",

abstract = "The US Department of Energy deployed the Summit and Sierra supercomputers with the latest state-of-the-art network interconnect technology in 2018 and both systems entered production in 2019. In this paper, we provide an in-depth assessment of the systems' network interconnects that are based on Enhanced Data Rate (EDR) 100 Gb/s Mellanox InfiniBand. Both systems use second-generation EDR Host Channel Adapters (HCAs) and switches with several new features such as Adaptive Routing (AR), switch-based collectives, and HCA-based tag matching. Although based on the same components, Summit's network is {"}non-blocking{"} (i.e., a fully provisioned Clos network) and Sierra's network has a 2:1 taper between the racks and aggregation switches. We evaluate the two systems' interconnects using traditional communication benchmarks as well as production applications. We find that the new Adaptive Routing dramatically improves performance but the other new features still need improvement.",

keywords = "Bandwidth, Congestion, EDR, High performance computing, InfiniBand, Interconnect, Latency, Offload, Switch collectives, Tag matching",

author = "Christopher Zimmer and Scott Atchley and Ramesh Pankajakshan and Smith, {Brian E.} and Ian Karlin and Leininger, {Matthew L.} and Adam Bertsch and Ryujin, {Brian S.} and Jason Burmark and Andr{\'e} Walker-Loud and Clark, {M. A.} and Olga Pearce",

note = "Publisher Copyright: {\textcopyright} 2019 ACM.; 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019 ; Conference date: 17-11-2019 Through 22-11-2019",

year = "2019",

month = nov,

day = "17",

doi = "10.1145/3295500.3356166",

language = "English",

series = "International Conference for High Performance Computing, Networking, Storage and Analysis, SC",

publisher = "IEEE Computer Society",

booktitle = "Proceedings of SC 2019",

}

Zimmer, C , Atchley, S, Pankajakshan, R, Smith, BE, Karlin, I, Leininger, ML, Bertsch, A, Ryujin, BS, Burmark, J, Walker-Loud, A, Clark, MA & Pearce, O 2019, An evaluation of the CORAL interconnects. in Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis., a39, International Conference for High Performance Computing, Networking, Storage and Analysis, SC, IEEE Computer Society, 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019, Denver, United States, 11/17/19. https://doi.org/10.1145/3295500.3356166

An evaluation of the CORAL interconnects. / Zimmer, Christopher ; Atchley, Scott; Pankajakshan, Ramesh et al.
Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society, 2019. a39 (International Conference for High Performance Computing, Networking, Storage and Analysis, SC).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - An evaluation of the CORAL interconnects

AU - Zimmer, Christopher

AU - Atchley, Scott

AU - Pankajakshan, Ramesh

AU - Smith, Brian E.

AU - Karlin, Ian

AU - Leininger, Matthew L.

AU - Bertsch, Adam

AU - Ryujin, Brian S.

AU - Burmark, Jason

AU - Walker-Loud, André

AU - Clark, M. A.

AU - Pearce, Olga

PY - 2019/11/17

Y1 - 2019/11/17

N2 - The US Department of Energy deployed the Summit and Sierra supercomputers with the latest state-of-the-art network interconnect technology in 2018 and both systems entered production in 2019. In this paper, we provide an in-depth assessment of the systems' network interconnects that are based on Enhanced Data Rate (EDR) 100 Gb/s Mellanox InfiniBand. Both systems use second-generation EDR Host Channel Adapters (HCAs) and switches with several new features such as Adaptive Routing (AR), switch-based collectives, and HCA-based tag matching. Although based on the same components, Summit's network is "non-blocking" (i.e., a fully provisioned Clos network) and Sierra's network has a 2:1 taper between the racks and aggregation switches. We evaluate the two systems' interconnects using traditional communication benchmarks as well as production applications. We find that the new Adaptive Routing dramatically improves performance but the other new features still need improvement.

AB - The US Department of Energy deployed the Summit and Sierra supercomputers with the latest state-of-the-art network interconnect technology in 2018 and both systems entered production in 2019. In this paper, we provide an in-depth assessment of the systems' network interconnects that are based on Enhanced Data Rate (EDR) 100 Gb/s Mellanox InfiniBand. Both systems use second-generation EDR Host Channel Adapters (HCAs) and switches with several new features such as Adaptive Routing (AR), switch-based collectives, and HCA-based tag matching. Although based on the same components, Summit's network is "non-blocking" (i.e., a fully provisioned Clos network) and Sierra's network has a 2:1 taper between the racks and aggregation switches. We evaluate the two systems' interconnects using traditional communication benchmarks as well as production applications. We find that the new Adaptive Routing dramatically improves performance but the other new features still need improvement.

KW - Bandwidth

KW - Congestion

KW - EDR

KW - High performance computing

KW - InfiniBand

KW - Interconnect

KW - Latency

KW - Offload

KW - Switch collectives

KW - Tag matching

UR - http://www.scopus.com/inward/record.url?scp=85076146055&partnerID=8YFLogxK

U2 - 10.1145/3295500.3356166

DO - 10.1145/3295500.3356166

M3 - Conference contribution

AN - SCOPUS:85076146055

T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC

BT - Proceedings of SC 2019

PB - IEEE Computer Society

T2 - 2019 International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2019

Y2 - 17 November 2019 through 22 November 2019

ER -

Zimmer C , Atchley S, Pankajakshan R, Smith BE, Karlin I, Leininger ML et al. An evaluation of the CORAL interconnects. In Proceedings of SC 2019: The International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE Computer Society. 2019. a39. (International Conference for High Performance Computing, Networking, Storage and Analysis, SC). doi: 10.1145/3295500.3356166

An evaluation of the CORAL interconnects

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this