A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform

Yawei Hui; Byung Hoon Park; Christian Engelmann

doi:10.1109/FTXS.2018.00007

A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform

Yawei Hui, Byung Hoon Park, Christian Engelmann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

4 Scopus citations

Abstract

Log processing by Spark and Cassandra-based ANalytics (LogSCAN) is a newly developed analytical platform that provides flexible and scalable data gathering, transformation and computation. One major challenge is to effectively summarize the status of a complex computer system, such as the Titan supercomputer at the Oak Ridge Leadership Computing Facility (OLCF). Although there is plenty of operational and maintenance information collected and stored in real time, which may yield insights about short- and long-term system status, it is difficult to present this information in a comprehensive form. In this work, we present system information entropy (SIE), a newly developed metric that leverages the powers of traditional machine learning techniques and information theory. By compressing the multivariant multi-dimensional event information recorded during the operation of the targeted system into a single time series of SIE, we demonstrate that the historical system status can be sensitively represented concisely and comprehensively. Given a sharp indicator as SIE, we argue that follow-up analytics based on SIE will reveal in-depth knowledge about system status using other sophisticated approaches, such as pattern recognition in the temporal domain or causality analysis incorporating extra independent metrics of the system.

Original language	English
Title of host publication	Proceedings of FTXS 2018
Subtitle of host publication	8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	39-48
Number of pages	10
ISBN (Electronic)	9781728102221
DOIs	https://doi.org/10.1109/FTXS.2018.00007
State	Published - Dec 5 2018
Event	8th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2018 - Dallas, United States Duration: Nov 11 2018 → Nov 16 2018

Publication series

Name	Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis

Conference

Conference	8th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2018
Country/Territory	United States
City	Dallas
Period	11/11/18 → 11/16/18

Funding

This manuscript has been authored by UT-Battelle,LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

Cloud-computing
Metrics
Visual-analytics

Access to Document

10.1109/FTXS.2018.00007

Cite this

Hui, Y., Park, B. H., & Engelmann, C. (2018). A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform. In Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis (pp. 39-48). Article 8564485 (Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/FTXS.2018.00007

Hui, Yawei ; Park, Byung Hoon ; Engelmann, Christian. / A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform. Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 39-48 (Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis).

@inproceedings{ab83781f0f03495e9938800e6e854266,

title = "A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform",

abstract = "Log processing by Spark and Cassandra-based ANalytics (LogSCAN) is a newly developed analytical platform that provides flexible and scalable data gathering, transformation and computation. One major challenge is to effectively summarize the status of a complex computer system, such as the Titan supercomputer at the Oak Ridge Leadership Computing Facility (OLCF). Although there is plenty of operational and maintenance information collected and stored in real time, which may yield insights about short- and long-term system status, it is difficult to present this information in a comprehensive form. In this work, we present system information entropy (SIE), a newly developed metric that leverages the powers of traditional machine learning techniques and information theory. By compressing the multivariant multi-dimensional event information recorded during the operation of the targeted system into a single time series of SIE, we demonstrate that the historical system status can be sensitively represented concisely and comprehensively. Given a sharp indicator as SIE, we argue that follow-up analytics based on SIE will reveal in-depth knowledge about system status using other sophisticated approaches, such as pattern recognition in the temporal domain or causality analysis incorporating extra independent metrics of the system.",

keywords = "Cloud-computing, Metrics, Visual-analytics",

author = "Yawei Hui and Park, {Byung Hoon} and Christian Engelmann",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 8th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2018 ; Conference date: 11-11-2018 Through 16-11-2018",

year = "2018",

month = dec,

day = "5",

doi = "10.1109/FTXS.2018.00007",

language = "English",

series = "Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "39--48",

booktitle = "Proceedings of FTXS 2018",

}

Hui, Y, Park, BH & Engelmann, C 2018, A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform. in Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis., 8564485, Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis, Institute of Electrical and Electronics Engineers Inc., pp. 39-48, 8th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2018, Dallas, United States, 11/11/18. https://doi.org/10.1109/FTXS.2018.00007

A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform. / Hui, Yawei; Park, Byung Hoon ; Engelmann, Christian.
Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc., 2018. p. 39-48 8564485 (Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform

AU - Hui, Yawei

AU - Park, Byung Hoon

AU - Engelmann, Christian

PY - 2018/12/5

Y1 - 2018/12/5

N2 - Log processing by Spark and Cassandra-based ANalytics (LogSCAN) is a newly developed analytical platform that provides flexible and scalable data gathering, transformation and computation. One major challenge is to effectively summarize the status of a complex computer system, such as the Titan supercomputer at the Oak Ridge Leadership Computing Facility (OLCF). Although there is plenty of operational and maintenance information collected and stored in real time, which may yield insights about short- and long-term system status, it is difficult to present this information in a comprehensive form. In this work, we present system information entropy (SIE), a newly developed metric that leverages the powers of traditional machine learning techniques and information theory. By compressing the multivariant multi-dimensional event information recorded during the operation of the targeted system into a single time series of SIE, we demonstrate that the historical system status can be sensitively represented concisely and comprehensively. Given a sharp indicator as SIE, we argue that follow-up analytics based on SIE will reveal in-depth knowledge about system status using other sophisticated approaches, such as pattern recognition in the temporal domain or causality analysis incorporating extra independent metrics of the system.

AB - Log processing by Spark and Cassandra-based ANalytics (LogSCAN) is a newly developed analytical platform that provides flexible and scalable data gathering, transformation and computation. One major challenge is to effectively summarize the status of a complex computer system, such as the Titan supercomputer at the Oak Ridge Leadership Computing Facility (OLCF). Although there is plenty of operational and maintenance information collected and stored in real time, which may yield insights about short- and long-term system status, it is difficult to present this information in a comprehensive form. In this work, we present system information entropy (SIE), a newly developed metric that leverages the powers of traditional machine learning techniques and information theory. By compressing the multivariant multi-dimensional event information recorded during the operation of the targeted system into a single time series of SIE, we demonstrate that the historical system status can be sensitively represented concisely and comprehensively. Given a sharp indicator as SIE, we argue that follow-up analytics based on SIE will reveal in-depth knowledge about system status using other sophisticated approaches, such as pattern recognition in the temporal domain or causality analysis incorporating extra independent metrics of the system.

KW - Cloud-computing

KW - Metrics

KW - Visual-analytics

UR - http://www.scopus.com/inward/record.url?scp=85060599310&partnerID=8YFLogxK

U2 - 10.1109/FTXS.2018.00007

DO - 10.1109/FTXS.2018.00007

M3 - Conference contribution

AN - SCOPUS:85060599310

T3 - Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis

SP - 39

EP - 48

BT - Proceedings of FTXS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 8th IEEE/ACM Workshop on Fault Tolerance for HPC at eXtreme Scale, FTXS 2018

Y2 - 11 November 2018 through 16 November 2018

ER -

Hui Y, Park BH , Engelmann C. A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform. In Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis. Institute of Electrical and Electronics Engineers Inc. 2018. p. 39-48. 8564485. (Proceedings of FTXS 2018: 8th Workshop on Fault Tolerance for HPC at eXtreme Scale, Held in conjunction with SC18: The International Conference for High Performance Computing, Networking, Storage and Analysis). doi: 10.1109/FTXS.2018.00007

A comprehensive informative metric for analyzing HPC system status using the LogSCAN platform

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this