Performance evaluation of python based data analytics frameworks in summit: Early experiences

Benjamín Hernández; Suhas Somnath; Junqi Yin; Hao Lu; Joe Eaton; Peter Entschev; John Kirkham; Zahra Ronaghi

doi:10.1007/978-3-030-63393-6_24

Performance evaluation of python based data analytics frameworks in summit: Early experiences

Benjamín Hernández
, Suhas Somnath
, Junqi Yin
, Hao Lu
, Joe Eaton
, Peter Entschev
, John Kirkham
, Zahra Ronaghi

Analytics and AI Methods at Scale

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

The explosion in the volumes of data generated from everlarger simulation campaigns and experiments or observations necessitates competent tools for data wrangling and analysis). While the Oak Ridge Leadership Computing Facility (OLCF) provides a variety of tools to perform data wrangling and data analysis tasks, Python based tools often lack scalability, or the ability to fully exploit the computational capability of OLCF’s Summit supercomputer. NVIDIA RAPIDS and Dask offer a promising solution to accelerate and distribute data analytics workloads from personal computers to heterogeneous supercomputing systems. We discuss early performance evaluation results of RAPIDS and Dask on Summit to understand their capabilities, scalability, and limitations. Our evaluation includes a subset of RAPIDS libraries, i.e., cuDF, cuML, and cuGraph, and Chainer’s CuPy, and their multi-GPU variants when available. We also draw on the observed trends from the performance evaluation results to discuss best practices for maximizing performance.

Original language	English
Title of host publication	Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers
Editors	Jeffrey Nichols, Arthur ‘Barney’ Maccabe, Suzanne Parete-Koon, Becky Verastegui, Oscar Hernandez, Theresa Ahearn
Publisher	Springer Science and Business Media Deutschland GmbH
Pages	366-380
Number of pages	15
ISBN (Print)	9783030633929
DOIs	https://doi.org/10.1007/978-3-030-63393-6_24
State	Published - 2021
Event	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 - Virtual, Online Duration: Aug 26 2020 → Aug 28 2020

Publication series

Name	Communications in Computer and Information Science
Volume	1315 CCIS
ISSN (Print)	1865-0929
ISSN (Electronic)	1865-0937

Conference

Conference	17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020
City	Virtual, Online
Period	08/26/20 → 08/28/20

Funding

B. Hernández et al.—Contributed Equally. This manuscript has been co-authored by UT-Battelle, LLC, under contract DE-AC05-00OR22725 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains a nonexclusive, paid-up, irrevocable, worldwide license to publish or reproduce the published form of this manuscript, or allow others to do so, for US government purposes. DOE will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy. gov/downloads/doe-public-access-plan). Acknowledgments. This research used resources of the Oak Ridge Leadership Computing Facility (OLCF) at the Oak Ridge National Laboratory, which is supported by the Office of Science of the U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Keywords

Data analytics
GPU
Multi-threaded
Performance evaluation
Python

Access to Document

10.1007/978-3-030-63393-6_24

Cite this

Hernández, B., Somnath, S., Yin, J., Lu, H., Eaton, J., Entschev, P., Kirkham, J., & Ronaghi, Z. (2021). Performance evaluation of python based data analytics frameworks in summit: Early experiences. In J. Nichols, A. B. Maccabe, S. Parete-Koon, B. Verastegui, O. Hernandez, & T. Ahearn (Eds.), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers (pp. 366-380). (Communications in Computer and Information Science; Vol. 1315 CCIS). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-030-63393-6_24

Hernández, Benjamín ; Somnath, Suhas ; Yin, Junqi et al. / Performance evaluation of python based data analytics frameworks in summit : Early experiences. Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. editor / Jeffrey Nichols ; Arthur ‘Barney’ Maccabe ; Suzanne Parete-Koon ; Becky Verastegui ; Oscar Hernandez ; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. pp. 366-380 (Communications in Computer and Information Science).

@inproceedings{bf5c1087fc70437aa87749ca5769d053,

title = "Performance evaluation of python based data analytics frameworks in summit: Early experiences",

abstract = "The explosion in the volumes of data generated from everlarger simulation campaigns and experiments or observations necessitates competent tools for data wrangling and analysis). While the Oak Ridge Leadership Computing Facility (OLCF) provides a variety of tools to perform data wrangling and data analysis tasks, Python based tools often lack scalability, or the ability to fully exploit the computational capability of OLCF{\textquoteright}s Summit supercomputer. NVIDIA RAPIDS and Dask offer a promising solution to accelerate and distribute data analytics workloads from personal computers to heterogeneous supercomputing systems. We discuss early performance evaluation results of RAPIDS and Dask on Summit to understand their capabilities, scalability, and limitations. Our evaluation includes a subset of RAPIDS libraries, i.e., cuDF, cuML, and cuGraph, and Chainer{\textquoteright}s CuPy, and their multi-GPU variants when available. We also draw on the observed trends from the performance evaluation results to discuss best practices for maximizing performance.",

keywords = "Data analytics, GPU, Multi-threaded, Performance evaluation, Python",

author = "Benjam{\'i}n Hern{\'a}ndez and Suhas Somnath and Junqi Yin and Hao Lu and Joe Eaton and Peter Entschev and John Kirkham and Zahra Ronaghi",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2020.; 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020 ; Conference date: 26-08-2020 Through 28-08-2020",

year = "2021",

doi = "10.1007/978-3-030-63393-6\_24",

language = "English",

isbn = "9783030633929",

series = "Communications in Computer and Information Science",

publisher = "Springer Science and Business Media Deutschland GmbH",

pages = "366--380",

editor = "Jeffrey Nichols and Maccabe, \{Arthur {\textquoteleft}Barney{\textquoteright}\} and Suzanne Parete-Koon and Becky Verastegui and Oscar Hernandez and Theresa Ahearn",

booktitle = "Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers",

}

Hernández, B, Somnath, S, Yin, J , Lu, H, Eaton, J, Entschev, P, Kirkham, J & Ronaghi, Z 2021, Performance evaluation of python based data analytics frameworks in summit: Early experiences. in J Nichols, AB Maccabe, S Parete-Koon, B Verastegui, O Hernandez & T Ahearn (eds), Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Communications in Computer and Information Science, vol. 1315 CCIS, Springer Science and Business Media Deutschland GmbH, pp. 366-380, 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Virtual, Online, 08/26/20. https://doi.org/10.1007/978-3-030-63393-6_24

Performance evaluation of python based data analytics frameworks in summit: Early experiences. / Hernández, Benjamín; Somnath, Suhas; Yin, Junqi et al.
Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. ed. / Jeffrey Nichols; Arthur ‘Barney’ Maccabe; Suzanne Parete-Koon; Becky Verastegui; Oscar Hernandez; Theresa Ahearn. Springer Science and Business Media Deutschland GmbH, 2021. p. 366-380 (Communications in Computer and Information Science; Vol. 1315 CCIS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Performance evaluation of python based data analytics frameworks in summit

T2 - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020

AU - Hernández, Benjamín

AU - Somnath, Suhas

AU - Yin, Junqi

AU - Lu, Hao

AU - Eaton, Joe

AU - Entschev, Peter

AU - Kirkham, John

AU - Ronaghi, Zahra

N1 - Publisher Copyright: © Springer Nature Switzerland AG 2020.

PY - 2021

Y1 - 2021

N2 - The explosion in the volumes of data generated from everlarger simulation campaigns and experiments or observations necessitates competent tools for data wrangling and analysis). While the Oak Ridge Leadership Computing Facility (OLCF) provides a variety of tools to perform data wrangling and data analysis tasks, Python based tools often lack scalability, or the ability to fully exploit the computational capability of OLCF’s Summit supercomputer. NVIDIA RAPIDS and Dask offer a promising solution to accelerate and distribute data analytics workloads from personal computers to heterogeneous supercomputing systems. We discuss early performance evaluation results of RAPIDS and Dask on Summit to understand their capabilities, scalability, and limitations. Our evaluation includes a subset of RAPIDS libraries, i.e., cuDF, cuML, and cuGraph, and Chainer’s CuPy, and their multi-GPU variants when available. We also draw on the observed trends from the performance evaluation results to discuss best practices for maximizing performance.

AB - The explosion in the volumes of data generated from everlarger simulation campaigns and experiments or observations necessitates competent tools for data wrangling and analysis). While the Oak Ridge Leadership Computing Facility (OLCF) provides a variety of tools to perform data wrangling and data analysis tasks, Python based tools often lack scalability, or the ability to fully exploit the computational capability of OLCF’s Summit supercomputer. NVIDIA RAPIDS and Dask offer a promising solution to accelerate and distribute data analytics workloads from personal computers to heterogeneous supercomputing systems. We discuss early performance evaluation results of RAPIDS and Dask on Summit to understand their capabilities, scalability, and limitations. Our evaluation includes a subset of RAPIDS libraries, i.e., cuDF, cuML, and cuGraph, and Chainer’s CuPy, and their multi-GPU variants when available. We also draw on the observed trends from the performance evaluation results to discuss best practices for maximizing performance.

KW - Data analytics

KW - GPU

KW - Multi-threaded

KW - Performance evaluation

KW - Python

UR - https://www.scopus.com/pages/publications/85107267115

U2 - 10.1007/978-3-030-63393-6_24

DO - 10.1007/978-3-030-63393-6_24

M3 - Conference contribution

AN - SCOPUS:85107267115

SN - 9783030633929

T3 - Communications in Computer and Information Science

SP - 366

EP - 380

BT - Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers

A2 - Nichols, Jeffrey

A2 - Maccabe, Arthur ‘Barney’

A2 - Parete-Koon, Suzanne

A2 - Verastegui, Becky

A2 - Hernandez, Oscar

A2 - Ahearn, Theresa

PB - Springer Science and Business Media Deutschland GmbH

Y2 - 26 August 2020 through 28 August 2020

ER -

Hernández B, Somnath S, Yin J , Lu H, Eaton J, Entschev P et al. Performance evaluation of python based data analytics frameworks in summit: Early experiences. In Nichols J, Maccabe AB, Parete-Koon S, Verastegui B, Hernandez O, Ahearn T, editors, Driving Scientific and Engineering Discoveries Through the Convergence of HPC, Big Data and AI - 17th Smoky Mountains Computational Sciences and Engineering Conference, SMC 2020, Revised Selected Papers. Springer Science and Business Media Deutschland GmbH. 2021. p. 366-380. (Communications in Computer and Information Science). doi: 10.1007/978-3-030-63393-6_24

Performance evaluation of python based data analytics frameworks in summit: Early experiences

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this