Understanding and analyzing interconnect errors and network congestion on a large scale HPC system

Mohit Kumar; Saurabh Gupta; Tirthak Patel; Michael Wilder; Weisong Shi; Song Fu; Christian Engelmann; Devesh Tiwari

doi:10.1109/DSN.2018.00023

Understanding and analyzing interconnect errors and network congestion on a large scale HPC system

Mohit Kumar, Saurabh Gupta, Tirthak Patel, Michael Wilder, Weisong Shi, Song Fu, Christian Engelmann, Devesh Tiwari

Intelligent Systems and Facilities

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

12 Scopus citations

Abstract

Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.

Original language	English
Title of host publication	Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	107-114
Number of pages	8
ISBN (Electronic)	9781538655955
DOIs	https://doi.org/10.1109/DSN.2018.00023
State	Published - Jul 19 2018
Event	48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 - Luxembourg City, Luxembourg Duration: Jun 25 2018 → Jun 28 2018

Publication series

Name	Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

Conference

Conference	48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018
Country/Territory	Luxembourg
City	Luxembourg City
Period	06/25/18 → 06/28/18

Funding

This manuscript has been authored by UT-Battelle,LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

Cray
Errors
Gemini
Interconnect
Titan

Access to Document

10.1109/DSN.2018.00023

Cite this

Kumar, M., Gupta, S., Patel, T., Wilder, M., Shi, W., Fu, S., Engelmann, C., & Tiwari, D. (2018). Understanding and analyzing interconnect errors and network congestion on a large scale HPC system. In Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 (pp. 107-114). Article 8416475 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/DSN.2018.00023

Kumar, Mohit ; Gupta, Saurabh ; Patel, Tirthak et al. / Understanding and analyzing interconnect errors and network congestion on a large scale HPC system. Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 107-114 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018).

@inproceedings{365434b257c7449184e9e9d807f4e8a3,

title = "Understanding and analyzing interconnect errors and network congestion on a large scale HPC system",

abstract = "Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.",

keywords = "Cray, Errors, Gemini, Interconnect, Titan",

author = "Mohit Kumar and Saurabh Gupta and Tirthak Patel and Michael Wilder and Weisong Shi and Song Fu and Christian Engelmann and Devesh Tiwari",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018 ; Conference date: 25-06-2018 Through 28-06-2018",

year = "2018",

month = jul,

day = "19",

doi = "10.1109/DSN.2018.00023",

language = "English",

series = "Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "107--114",

booktitle = "Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018",

}

Kumar, M, Gupta, S, Patel, T, Wilder, M, Shi, W, Fu, S, Engelmann, C & Tiwari, D 2018, Understanding and analyzing interconnect errors and network congestion on a large scale HPC system. in Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018., 8416475, Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Institute of Electrical and Electronics Engineers Inc., pp. 107-114, 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018, Luxembourg City, Luxembourg, 06/25/18. https://doi.org/10.1109/DSN.2018.00023

Understanding and analyzing interconnect errors and network congestion on a large scale HPC system. / Kumar, Mohit; Gupta, Saurabh; Patel, Tirthak et al.
Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 107-114 8416475 (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Understanding and analyzing interconnect errors and network congestion on a large scale HPC system

AU - Kumar, Mohit

AU - Gupta, Saurabh

AU - Patel, Tirthak

AU - Wilder, Michael

AU - Shi, Weisong

AU - Fu, Song

AU - Engelmann, Christian

AU - Tiwari, Devesh

PY - 2018/7/19

Y1 - 2018/7/19

N2 - Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.

AB - Today's High Performance Computing (HPC) systems are capable of delivering performance in the order of petaflops due to the fast computing devices, network interconnect, and back-end storage systems. In particular, interconnect resilience and congestion resolution methods have a major impact on the overall interconnect and application performance. This is especially true for scientific applications running multiple processes on different compute nodes as they rely on fast network messages to communicate and synchronize frequently. Unfortunately, the HPC community lacks state-of-practice experience reports that detail how different interconnect errors and congestion events occur on large-scale HPC systems. Therefore, in this paper, we process and analyze interconnect data of the Titan supercomputer to develop a thorough understanding of interconnects faults, errors and congestion events. We also study the interaction between interconnect, errors, network congestion and application characteristics.

KW - Cray

KW - Errors

KW - Gemini

KW - Interconnect

KW - Titan

UR - http://www.scopus.com/inward/record.url?scp=85051065059&partnerID=8YFLogxK

U2 - 10.1109/DSN.2018.00023

DO - 10.1109/DSN.2018.00023

M3 - Conference contribution

AN - SCOPUS:85051065059

T3 - Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

SP - 107

EP - 114

BT - Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018

Y2 - 25 June 2018 through 28 June 2018

ER -

Kumar M, Gupta S, Patel T, Wilder M, Shi W, Fu S et al. Understanding and analyzing interconnect errors and network congestion on a large scale HPC system. In Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 107-114. 8416475. (Proceedings - 48th Annual IEEE/IFIP International Conference on Dependable Systems and Networks, DSN 2018). doi: 10.1109/DSN.2018.00023

Understanding and analyzing interconnect errors and network congestion on a large scale HPC system

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this