A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Byung H. Park; Yawei Hui; Swen Boehm; Rizwan A. Ashraf; Christopher Layton; Christian Engelmann

doi:10.1109/CLUSTER.2018.00073

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Byung H. Park, Yawei Hui, Swen Boehm, Rizwan A. Ashraf, Christopher Layton, Christian Engelmann

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

13 Scopus citations

Abstract

Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.

Original language	English
Title of host publication	Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	571-579
Number of pages	9
ISBN (Electronic)	9781538683194
DOIs	https://doi.org/10.1109/CLUSTER.2018.00073
State	Published - Oct 29 2018
Event	2018 IEEE International Conference on Cluster Computing, CLUSTER 2018 - Belfast, United Kingdom Duration: Sep 10 2018 → Sep 13 2018

Publication series

Name	Proceedings - IEEE International Conference on Cluster Computing, ICCC
Volume	2018-September
ISSN (Print)	1552-5244

Conference

Conference	2018 IEEE International Conference on Cluster Computing, CLUSTER 2018
Country/Territory	United Kingdom
City	Belfast
Period	09/10/18 → 09/13/18

Funding

This manuscript has been authored by UT-Battelle,LLC under Contract No. DE-AC05-00OR22725with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

Big Data applications
Data analysis
Event log analysis
High performance computing

Access to Document

10.1109/CLUSTER.2018.00073

Cite this

Park, B. H., Hui, Y., Boehm, S., Ashraf, R. A., Layton, C., & Engelmann, C. (2018). A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log. In Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018 (pp. 571-579). Article 8514917 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2018-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2018.00073

Park, Byung H. ; Hui, Yawei ; Boehm, Swen et al. / A Big Data Analytics Framework for HPC Log Data : Three Case Studies Using the Titan Supercomputer Log. Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 571-579 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).

@inproceedings{4bacddb41e16416ba9098c11bb65816d,

title = "A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log",

abstract = "Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.",

keywords = "Big Data applications, Data analysis, Event log analysis, High performance computing",

author = "Park, {Byung H.} and Yawei Hui and Swen Boehm and Ashraf, {Rizwan A.} and Christopher Layton and Christian Engelmann",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018 ; Conference date: 10-09-2018 Through 13-09-2018",

year = "2018",

month = oct,

day = "29",

doi = "10.1109/CLUSTER.2018.00073",

language = "English",

series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "571--579",

booktitle = "Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018",

}

Park, BH, Hui, Y, Boehm, S, Ashraf, RA, Layton, C & Engelmann, C 2018, A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log. in Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018., 8514917, Proceedings - IEEE International Conference on Cluster Computing, ICCC, vol. 2018-September, Institute of Electrical and Electronics Engineers Inc., pp. 571-579, 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018, Belfast, United Kingdom, 09/10/18. https://doi.org/10.1109/CLUSTER.2018.00073

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log. / Park, Byung H.; Hui, Yawei; Boehm, Swen et al.
Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 571-579 8514917 (Proceedings - IEEE International Conference on Cluster Computing, ICCC; Vol. 2018-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A Big Data Analytics Framework for HPC Log Data

T2 - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018

AU - Park, Byung H.

AU - Hui, Yawei

AU - Boehm, Swen

AU - Ashraf, Rizwan A.

AU - Layton, Christopher

AU - Engelmann, Christian

PY - 2018/10/29

Y1 - 2018/10/29

N2 - Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.

AB - Reliability, availability and serviceability (RAS) logs of high performance computing (HPC) resources, when closely investigated in spatial and temporal dimensions, can provide invaluable information regarding system status, performance, and resource utilization. These data are often generated from multiple logging systems and sensors that cover many components of the system. The analysis of these data for finding persistent temporal and spatial insights faces two main difficulties: the volume of RAS logs makes manual inspection difficult and the unstructured nature and unique properties of log data produced by each subsystem adds another dimension of difficulty in identifying implicit correlation among recorded events. To address these issues, we recently developed a multi-user Big Data analytics framework for HPC log data at Oak Ridge National Laboratory (ORNL). This paper introduces three in-progress data analytics projects that leverage this framework to assess system status, mine event patterns, and study correlations between user applications and system events. We describe the motivation of each project and detail their workflows using three years of log data collected from ORNL's Titan supercomputer.

KW - Big Data applications

KW - Data analysis

KW - Event log analysis

KW - High performance computing

UR - http://www.scopus.com/inward/record.url?scp=85057269230&partnerID=8YFLogxK

U2 - 10.1109/CLUSTER.2018.00073

DO - 10.1109/CLUSTER.2018.00073

M3 - Conference contribution

AN - SCOPUS:85057269230

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

SP - 571

EP - 579

BT - Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018

PB - Institute of Electrical and Electronics Engineers Inc.

Y2 - 10 September 2018 through 13 September 2018

ER -

Park BH, Hui Y, Boehm S, Ashraf RA, Layton C, Engelmann C. A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log. In Proceedings - 2018 IEEE International Conference on Cluster Computing, CLUSTER 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 571-579. 8514917. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). doi: 10.1109/CLUSTER.2018.00073

A Big Data Analytics Framework for HPC Log Data: Three Case Studies Using the Titan Supercomputer Log

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this