Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility

Sangkeun Lee; Sudharshan S. Vazhkudai; Raghul Gunasekaran

doi:10.1109/HiPC.2017.00041

Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility

Sangkeun Lee, Sudharshan S. Vazhkudai, Raghul Gunasekaran

Discrete Algorithms

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

Abstract

The Oak Ridge Leadership Computing Facility (OLCF) runs Titan, the No. 4 supercomputer in the world, to deliver over four billion compute core hours every year to several scientific domains, in their pursuit of leadership science. In this paper, we analyze four years worth of heterogeneous log data sources from the OLCF resource fabric, capturing metadata on entities such as users (2,546), scientific project allocations (674), jobs (1,352,402) and publications (1,146), to derive insights into the trends in core hour usage and publications, across 35 science domains. We have constructed a scalable graph to represent the OLCF entities and apply rich graph analytics for our analysis. Based on this, we have analyzed the metadata across five dimensions, namely (1) quantitative analysis of Titan system usage, (2) quantitative analysis of OLCF publications, (3) correlation analysis between system usage and publications, (4) text analysis to derive OLCF research trends, and (5) utilization of graph mining for association analysis. To the best of our knowledge, our work is the first of its kind to apply graph- based big data techniques to provide comprehensive insights on an HPC center's core hour usage and users' publication trends. Our results provide valuable details into an HPC center's core allocation program, measuring the productivity of scientific domains, the interplay between core usage and research output, accelerating collaboration, and in predicting new connections between resource entities.

Original language	English
Title of host publication	Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	294-305
Number of pages	12
ISBN (Electronic)	9781538622933
DOIs	https://doi.org/10.1109/HiPC.2017.00041
State	Published - Jul 2 2017
Event	24th IEEE International Conference on High Performance Computing, HiPC 2017 - Jaipur, India Duration: Dec 18 2017 → Dec 21 2017

Publication series

Name	Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017
Volume	2017-December

Conference

Conference	24th IEEE International Conference on High Performance Computing, HiPC 2017
Country/Territory	India
City	Jaipur
Period	12/18/17 → 12/21/17

Funding

This manuscript has been authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paidup, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan).

Keywords

HPC
OLCF
Supercomputing
graph analysis
log analysis

Access to Document

10.1109/HiPC.2017.00041

Cite this

Lee, S., Vazhkudai, S. S., & Gunasekaran, R. (2017). Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility. In Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017 (pp. 294-305). (Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017; Vol. 2017-December). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/HiPC.2017.00041

Lee, Sangkeun ; Vazhkudai, Sudharshan S. ; Gunasekaran, Raghul. / Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility. Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017. Institute of Electrical and Electronics Engineers Inc., 2017. pp. 294-305 (Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017).

@inproceedings{ea4327da28f946afb9ccb909fb0fa74b,

title = "Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility",

abstract = "The Oak Ridge Leadership Computing Facility (OLCF) runs Titan, the No. 4 supercomputer in the world, to deliver over four billion compute core hours every year to several scientific domains, in their pursuit of leadership science. In this paper, we analyze four years worth of heterogeneous log data sources from the OLCF resource fabric, capturing metadata on entities such as users (2,546), scientific project allocations (674), jobs (1,352,402) and publications (1,146), to derive insights into the trends in core hour usage and publications, across 35 science domains. We have constructed a scalable graph to represent the OLCF entities and apply rich graph analytics for our analysis. Based on this, we have analyzed the metadata across five dimensions, namely (1) quantitative analysis of Titan system usage, (2) quantitative analysis of OLCF publications, (3) correlation analysis between system usage and publications, (4) text analysis to derive OLCF research trends, and (5) utilization of graph mining for association analysis. To the best of our knowledge, our work is the first of its kind to apply graph- based big data techniques to provide comprehensive insights on an HPC center's core hour usage and users' publication trends. Our results provide valuable details into an HPC center's core allocation program, measuring the productivity of scientific domains, the interplay between core usage and research output, accelerating collaboration, and in predicting new connections between resource entities.",

keywords = "HPC, OLCF, Supercomputing, graph analysis, log analysis",

author = "Sangkeun Lee and Vazhkudai, {Sudharshan S.} and Raghul Gunasekaran",

note = "Publisher Copyright: {\textcopyright} 2017 IEEE.; 24th IEEE International Conference on High Performance Computing, HiPC 2017 ; Conference date: 18-12-2017 Through 21-12-2017",

year = "2017",

month = jul,

day = "2",

doi = "10.1109/HiPC.2017.00041",

language = "English",

series = "Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "294--305",

booktitle = "Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017",

}

Lee, S, Vazhkudai, SS & Gunasekaran, R 2017, Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility. in Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017. Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017, vol. 2017-December, Institute of Electrical and Electronics Engineers Inc., pp. 294-305, 24th IEEE International Conference on High Performance Computing, HiPC 2017, Jaipur, India, 12/18/17. https://doi.org/10.1109/HiPC.2017.00041

Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility. / Lee, Sangkeun; Vazhkudai, Sudharshan S.; Gunasekaran, Raghul.
Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017. Institute of Electrical and Electronics Engineers Inc., 2017. p. 294-305 (Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017; Vol. 2017-December).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility

AU - Lee, Sangkeun

AU - Vazhkudai, Sudharshan S.

AU - Gunasekaran, Raghul

PY - 2017/7/2

Y1 - 2017/7/2

N2 - The Oak Ridge Leadership Computing Facility (OLCF) runs Titan, the No. 4 supercomputer in the world, to deliver over four billion compute core hours every year to several scientific domains, in their pursuit of leadership science. In this paper, we analyze four years worth of heterogeneous log data sources from the OLCF resource fabric, capturing metadata on entities such as users (2,546), scientific project allocations (674), jobs (1,352,402) and publications (1,146), to derive insights into the trends in core hour usage and publications, across 35 science domains. We have constructed a scalable graph to represent the OLCF entities and apply rich graph analytics for our analysis. Based on this, we have analyzed the metadata across five dimensions, namely (1) quantitative analysis of Titan system usage, (2) quantitative analysis of OLCF publications, (3) correlation analysis between system usage and publications, (4) text analysis to derive OLCF research trends, and (5) utilization of graph mining for association analysis. To the best of our knowledge, our work is the first of its kind to apply graph- based big data techniques to provide comprehensive insights on an HPC center's core hour usage and users' publication trends. Our results provide valuable details into an HPC center's core allocation program, measuring the productivity of scientific domains, the interplay between core usage and research output, accelerating collaboration, and in predicting new connections between resource entities.

AB - The Oak Ridge Leadership Computing Facility (OLCF) runs Titan, the No. 4 supercomputer in the world, to deliver over four billion compute core hours every year to several scientific domains, in their pursuit of leadership science. In this paper, we analyze four years worth of heterogeneous log data sources from the OLCF resource fabric, capturing metadata on entities such as users (2,546), scientific project allocations (674), jobs (1,352,402) and publications (1,146), to derive insights into the trends in core hour usage and publications, across 35 science domains. We have constructed a scalable graph to represent the OLCF entities and apply rich graph analytics for our analysis. Based on this, we have analyzed the metadata across five dimensions, namely (1) quantitative analysis of Titan system usage, (2) quantitative analysis of OLCF publications, (3) correlation analysis between system usage and publications, (4) text analysis to derive OLCF research trends, and (5) utilization of graph mining for association analysis. To the best of our knowledge, our work is the first of its kind to apply graph- based big data techniques to provide comprehensive insights on an HPC center's core hour usage and users' publication trends. Our results provide valuable details into an HPC center's core allocation program, measuring the productivity of scientific domains, the interplay between core usage and research output, accelerating collaboration, and in predicting new connections between resource entities.

KW - HPC

KW - OLCF

KW - Supercomputing

KW - graph analysis

KW - log analysis

UR - http://www.scopus.com/inward/record.url?scp=85050354885&partnerID=8YFLogxK

U2 - 10.1109/HiPC.2017.00041

DO - 10.1109/HiPC.2017.00041

M3 - Conference contribution

AN - SCOPUS:85050354885

T3 - Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017

SP - 294

EP - 305

BT - Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 24th IEEE International Conference on High Performance Computing, HiPC 2017

Y2 - 18 December 2017 through 21 December 2017

ER -

Lee S, Vazhkudai SS, Gunasekaran R. Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility. In Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017. Institute of Electrical and Electronics Engineers Inc. 2017. p. 294-305. (Proceedings - 24th IEEE International Conference on High Performance Computing, HiPC 2017). doi: 10.1109/HiPC.2017.00041

Applying Graph Analytics to Understand Compute Core Usage and Publication Trends in a Petascale Supercomputing Facility

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this