Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing

Yawei Hui; Rizwan A. Ashraf; Byung H. Park; Christian Engelmann

doi:10.1109/BigData.2018.8621862

Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing

Yawei Hui, Rizwan A. Ashraf, Byung H. Park, Christian Engelmann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

Supercomputers are complex systems used to simulate, understand and solve real-world problems. In order to operate these systems efficiently and for the purpose of their maintainability, an accurate, concise, and timely determination of system status is crucial for its users and operators. However, this determination is challenging due to intricately connected heterogeneous software and hardware components, and due to sheer scale of such machines. In this poster, we demonstrate work-in-progress towards realization of a real-time monitoring framework for the 18,688-node Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). Toward this end, we discuss the use of metrics which present a one-dimensional view of the system generating various types of information from 1000s of components and utilization statistics from 100s of user applications in near real-time. We demonstrate the efficacy of these metrics to understand and visualize raw log data generated by the system which otherwise may compose of 1000s of dimensions. We also demonstrate the architecture of proposed real-time stream processing framework which integrates, processes, analyzes, visualizes and stores system log data from an array of system components.

Original language	English
Title of host publication	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018
Editors	Naoki Abe, Huan Liu, Calton Pu, Xiaohua Hu, Nesreen Ahmed, Mu Qiao, Yang Song, Donald Kossmann, Bing Liu, Kisung Lee, Jiliang Tang, Jingrui He, Jeffrey Saltz
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	5339-5341
Number of pages	3
ISBN (Electronic)	9781538650356
DOIs	https://doi.org/10.1109/BigData.2018.8621862
State	Published - Jul 2 2018
Event	2018 IEEE International Conference on Big Data, Big Data 2018 - Seattle, United States Duration: Dec 10 2018 → Dec 13 2018

Publication series

Name	Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

Conference

Conference	2018 IEEE International Conference on Big Data, Big Data 2018
Country/Territory	United States
City	Seattle
Period	12/10/18 → 12/13/18

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of Advanced Scientific Computing Research, Resilience for Extreme Scale Supercomputing Systems Program, with program manager Lucy Nowell, under contract number DE-AC05-00OR22725.

Keywords

Availability
Quantification Metrics
Reliability
Serviceability
System Monitoring

Access to Document

10.1109/BigData.2018.8621862

Cite this

Hui, Y., Ashraf, R. A., Park, B. H., & Engelmann, C. (2018). Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing. In N. Abe, H. Liu, C. Pu, X. Hu, N. Ahmed, M. Qiao, Y. Song, D. Kossmann, B. Liu, K. Lee, J. Tang, J. He, & J. Saltz (Eds.), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018 (pp. 5339-5341). Article 8621862 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/BigData.2018.8621862

Hui, Yawei ; Ashraf, Rizwan A. ; Park, Byung H. et al. / Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing. Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. editor / Naoki Abe ; Huan Liu ; Calton Pu ; Xiaohua Hu ; Nesreen Ahmed ; Mu Qiao ; Yang Song ; Donald Kossmann ; Bing Liu ; Kisung Lee ; Jiliang Tang ; Jingrui He ; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 5339-5341 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

@inproceedings{ad81e630999449a49bec1c7fba57a07c,

title = "Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing",

abstract = "Supercomputers are complex systems used to simulate, understand and solve real-world problems. In order to operate these systems efficiently and for the purpose of their maintainability, an accurate, concise, and timely determination of system status is crucial for its users and operators. However, this determination is challenging due to intricately connected heterogeneous software and hardware components, and due to sheer scale of such machines. In this poster, we demonstrate work-in-progress towards realization of a real-time monitoring framework for the 18,688-node Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). Toward this end, we discuss the use of metrics which present a one-dimensional view of the system generating various types of information from 1000s of components and utilization statistics from 100s of user applications in near real-time. We demonstrate the efficacy of these metrics to understand and visualize raw log data generated by the system which otherwise may compose of 1000s of dimensions. We also demonstrate the architecture of proposed real-time stream processing framework which integrates, processes, analyzes, visualizes and stores system log data from an array of system components.",

keywords = "Availability, Quantification Metrics, Reliability, Serviceability, System Monitoring",

author = "Yawei Hui and Ashraf, {Rizwan A.} and Park, {Byung H.} and Christian Engelmann",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 IEEE International Conference on Big Data, Big Data 2018 ; Conference date: 10-12-2018 Through 13-12-2018",

year = "2018",

month = jul,

day = "2",

doi = "10.1109/BigData.2018.8621862",

language = "English",

series = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "5339--5341",

editor = "Naoki Abe and Huan Liu and Calton Pu and Xiaohua Hu and Nesreen Ahmed and Mu Qiao and Yang Song and Donald Kossmann and Bing Liu and Kisung Lee and Jiliang Tang and Jingrui He and Jeffrey Saltz",

booktitle = "Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018",

}

Hui, Y, Ashraf, RA, Park, BH & Engelmann, C 2018, Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing. in N Abe, H Liu, C Pu, X Hu, N Ahmed, M Qiao, Y Song, D Kossmann, B Liu, K Lee, J Tang, J He & J Saltz (eds), Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018., 8621862, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018, Institute of Electrical and Electronics Engineers Inc., pp. 5339-5341, 2018 IEEE International Conference on Big Data, Big Data 2018, Seattle, United States, 12/10/18. https://doi.org/10.1109/BigData.2018.8621862

Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing. / Hui, Yawei; Ashraf, Rizwan A.; Park, Byung H. et al.
Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. ed. / Naoki Abe; Huan Liu; Calton Pu; Xiaohua Hu; Nesreen Ahmed; Mu Qiao; Yang Song; Donald Kossmann; Bing Liu; Kisung Lee; Jiliang Tang; Jingrui He; Jeffrey Saltz. Institute of Electrical and Electronics Engineers Inc., 2018. p. 5339-5341 8621862 (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing

AU - Hui, Yawei

AU - Ashraf, Rizwan A.

AU - Park, Byung H.

AU - Engelmann, Christian

PY - 2018/7/2

Y1 - 2018/7/2

N2 - Supercomputers are complex systems used to simulate, understand and solve real-world problems. In order to operate these systems efficiently and for the purpose of their maintainability, an accurate, concise, and timely determination of system status is crucial for its users and operators. However, this determination is challenging due to intricately connected heterogeneous software and hardware components, and due to sheer scale of such machines. In this poster, we demonstrate work-in-progress towards realization of a real-time monitoring framework for the 18,688-node Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). Toward this end, we discuss the use of metrics which present a one-dimensional view of the system generating various types of information from 1000s of components and utilization statistics from 100s of user applications in near real-time. We demonstrate the efficacy of these metrics to understand and visualize raw log data generated by the system which otherwise may compose of 1000s of dimensions. We also demonstrate the architecture of proposed real-time stream processing framework which integrates, processes, analyzes, visualizes and stores system log data from an array of system components.

AB - Supercomputers are complex systems used to simulate, understand and solve real-world problems. In order to operate these systems efficiently and for the purpose of their maintainability, an accurate, concise, and timely determination of system status is crucial for its users and operators. However, this determination is challenging due to intricately connected heterogeneous software and hardware components, and due to sheer scale of such machines. In this poster, we demonstrate work-in-progress towards realization of a real-time monitoring framework for the 18,688-node Titan supercomputer at Oak Ridge Leadership Computing Facility (OLCF). Toward this end, we discuss the use of metrics which present a one-dimensional view of the system generating various types of information from 1000s of components and utilization statistics from 100s of user applications in near real-time. We demonstrate the efficacy of these metrics to understand and visualize raw log data generated by the system which otherwise may compose of 1000s of dimensions. We also demonstrate the architecture of proposed real-time stream processing framework which integrates, processes, analyzes, visualizes and stores system log data from an array of system components.

KW - Availability

KW - Quantification Metrics

KW - Reliability

KW - Serviceability

KW - System Monitoring

UR - http://www.scopus.com/inward/record.url?scp=85062608809&partnerID=8YFLogxK

U2 - 10.1109/BigData.2018.8621862

DO - 10.1109/BigData.2018.8621862

M3 - Conference contribution

AN - SCOPUS:85062608809

T3 - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

SP - 5339

EP - 5341

BT - Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018

A2 - Abe, Naoki

A2 - Liu, Huan

A2 - Pu, Calton

A2 - Hu, Xiaohua

A2 - Ahmed, Nesreen

A2 - Qiao, Mu

A2 - Song, Yang

A2 - Kossmann, Donald

A2 - Liu, Bing

A2 - Lee, Kisung

A2 - Tang, Jiliang

A2 - He, Jingrui

A2 - Saltz, Jeffrey

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2018 IEEE International Conference on Big Data, Big Data 2018

Y2 - 10 December 2018 through 13 December 2018

ER -

Hui Y, Ashraf RA, Park BH , Engelmann C. Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing. In Abe N, Liu H, Pu C, Hu X, Ahmed N, Qiao M, Song Y, Kossmann D, Liu B, Lee K, Tang J, He J, Saltz J, editors, Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 5339-5341. 8621862. (Proceedings - 2018 IEEE International Conference on Big Data, Big Data 2018). doi: 10.1109/BigData.2018.8621862

Real-Time Assessment of Supercomputer Status by a Comprehensive Informative Metric through Streaming Processing

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this