Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer

Giovanni Balduzzi; Arghya Chatterjee; Ying Wai Li; Peter W. Doak; Urs Haehner; Ed F. D'Azevedo; Thomas A. Maier; Thomas Schulthess

doi:10.1109/PACT.2019.00041

Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer

Giovanni Balduzzi, Arghya Chatterjee, Ying Wai Li, Peter W. Doak, Urs Haehner, Ed F. D'Azevedo, Thomas A. Maier, Thomas Schulthess

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

5 Scopus citations

Abstract

Optimizing scientific applications on today's accelerator-based high performance computing systems can be challenging, especially when multiple GPUs and CPUs with heterogeneous memories and persistent non-volatile memories are present. An example is Summit, an accelerator-based system at the Oak Ridge Leadership Computing Facility (OLCF) that is rated as the world's fastest supercomputer to-date. New strategies are thus needed to expose the parallelism in legacy applications, while being amenable to efficient mapping to the underlying architecture. In this paper we discuss our experiences and strategies to port a scientific application, DCA++, to Summit. DCA++ is a high-performance research application that solves quantum many-body problems with a cutting edge quantum cluster algorithm, the dynamical cluster approximation. Our strategies aim to synergize the strengths of the different programming models in the code. These include: A) streamlining the interactions between the CPU threads and the GPUs, b) implementing computing kernels on the GPUs and decreasing CPU-GPU memory transfers, c) allowing asynchronous GPU communications, and d) increasing compute intensity by combining linear algebraic operations. Full-scale production runs using all 4600 Summit nodes attained a peak performance of 73.5 PFLOPS with a mixed precision implementation. We observed a perfect strong and weak scaling for the quantum Monte Carlo solver in DCA++, while encountering about 2x input/output (I/O) and MPI communication overhead on the time-To-solution for the full machine run. Our hardware agnostic optimizations are designed to alleviate the communication and I/O challenges observed, while improving the compute intensity and obtaining optimal performance on a complex, hybrid architecture like Summit.

Original language	English
Title of host publication	Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	432-443
Number of pages	12
ISBN (Electronic)	9781728136134
DOIs	https://doi.org/10.1109/PACT.2019.00041
State	Published - Sep 2019
Event	28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019 - Seattle, United States Duration: Sep 21 2019 → Sep 25 2019

Publication series

Name	Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT
Volume	2019-September
ISSN (Print)	1089-795X

Conference

Conference	28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019
Country/Territory	United States
City	Seattle
Period	09/21/19 → 09/25/19

Funding

This manuscript has been co-authored by UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the U.S. Department of Energy. The United States Government retains and the publisher, by accepting the article for publication, acknowledges that the United States Government retains a non-exclusive, paid-up, irrevocable, world-wide license to publish or reproduce the published form of this manuscript, or allow others to do so, for United States Government purposes. The Department of Energy will provide public access to these results of federally sponsored research in accordance with the DOE Public Access Plan (http://energy.gov/downloads/doe-public-access-plan). ¶All three authors have equal contribution to the manuscript. † Ying Wai Li contributed to this work mostly during her previous appointment at Oak Ridge National Laboratory, Oak Ridge 37831, U.S. Authors would like to thank Oscar Hernandez (ORNL), Jeff Larkin (NVIDIA), Don Maxwell (ORNL), Ronny Bren-del (Score-P), John Mellor-Crummey (HPCToolkit) for their insights during the optimization phase of DCA++. This work was supported by the Scientific Discovery through Advanced Computing (SciDAC) program funded by U.S. Department of Energy, Office of Science, Advanced Scientific Computing Research and Basic Energy Sciences, Division of Materials Sciences and Engineering. This research used resources of the Oak Ridge Leadership Computing Facility, which is a DOE Office of Science User Facility supported under Contract DE-AC05-00OR22725.

Funders	Funder number
DOE Office of Science User Facility supported	DE-AC05-00OR22725
LLC
Oak
Scientific Discovery
UT-Battelle
U.S. Department of Energy
Office of Science
Advanced Scientific Computing Research
Division of Materials Sciences and Engineering

Keywords

CUDA
CUDA aware MPI
DCA
QMC
Quantum Monte Carlo
Spectrum MPI
Summit@OLCF

Access to Document

10.1109/PACT.2019.00041

Cite this

Balduzzi, G., Chatterjee, A., Li, Y. W., Doak, P. W., Haehner, U., D'Azevedo, E. F., Maier, T. A., & Schulthess, T. (2019). Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer. In Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019 (pp. 432-443). Article 8891643 (Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT; Vol. 2019-September). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/PACT.2019.00041

Balduzzi, Giovanni ; Chatterjee, Arghya ; Li, Ying Wai et al. / Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer. Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019. Institute of Electrical and Electronics Engineers Inc., 2019. pp. 432-443 (Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT).

@inproceedings{c6f1fe77a42e444992795a3fd6fd9131,

title = "Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer",

abstract = "Optimizing scientific applications on today's accelerator-based high performance computing systems can be challenging, especially when multiple GPUs and CPUs with heterogeneous memories and persistent non-volatile memories are present. An example is Summit, an accelerator-based system at the Oak Ridge Leadership Computing Facility (OLCF) that is rated as the world's fastest supercomputer to-date. New strategies are thus needed to expose the parallelism in legacy applications, while being amenable to efficient mapping to the underlying architecture. In this paper we discuss our experiences and strategies to port a scientific application, DCA++, to Summit. DCA++ is a high-performance research application that solves quantum many-body problems with a cutting edge quantum cluster algorithm, the dynamical cluster approximation. Our strategies aim to synergize the strengths of the different programming models in the code. These include: A) streamlining the interactions between the CPU threads and the GPUs, b) implementing computing kernels on the GPUs and decreasing CPU-GPU memory transfers, c) allowing asynchronous GPU communications, and d) increasing compute intensity by combining linear algebraic operations. Full-scale production runs using all 4600 Summit nodes attained a peak performance of 73.5 PFLOPS with a mixed precision implementation. We observed a perfect strong and weak scaling for the quantum Monte Carlo solver in DCA++, while encountering about 2x input/output (I/O) and MPI communication overhead on the time-To-solution for the full machine run. Our hardware agnostic optimizations are designed to alleviate the communication and I/O challenges observed, while improving the compute intensity and obtaining optimal performance on a complex, hybrid architecture like Summit.",

keywords = "CUDA, CUDA aware MPI, DCA, QMC, Quantum Monte Carlo, Spectrum MPI, Summit@OLCF",

author = "Giovanni Balduzzi and Arghya Chatterjee and Li, {Ying Wai} and Doak, {Peter W.} and Urs Haehner and D'Azevedo, {Ed F.} and Maier, {Thomas A.} and Thomas Schulthess",

note = "Publisher Copyright: {\textcopyright} 2019 IEEE.; 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019 ; Conference date: 21-09-2019 Through 25-09-2019",

year = "2019",

month = sep,

doi = "10.1109/PACT.2019.00041",

language = "English",

series = "Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "432--443",

booktitle = "Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019",

}

Balduzzi, G, Chatterjee, A, Li, YW, Doak, PW, Haehner, U, D'Azevedo, EF, Maier, TA & Schulthess, T 2019, Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer. in Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019., 8891643, Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT, vol. 2019-September, Institute of Electrical and Electronics Engineers Inc., pp. 432-443, 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019, Seattle, United States, 09/21/19. https://doi.org/10.1109/PACT.2019.00041

Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer. / Balduzzi, Giovanni; Chatterjee, Arghya; Li, Ying Wai et al.
Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019. Institute of Electrical and Electronics Engineers Inc., 2019. p. 432-443 8891643 (Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT; Vol. 2019-September).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer

AU - Balduzzi, Giovanni

AU - Chatterjee, Arghya

AU - Li, Ying Wai

AU - Doak, Peter W.

AU - Haehner, Urs

AU - D'Azevedo, Ed F.

AU - Maier, Thomas A.

AU - Schulthess, Thomas

PY - 2019/9

Y1 - 2019/9

N2 - Optimizing scientific applications on today's accelerator-based high performance computing systems can be challenging, especially when multiple GPUs and CPUs with heterogeneous memories and persistent non-volatile memories are present. An example is Summit, an accelerator-based system at the Oak Ridge Leadership Computing Facility (OLCF) that is rated as the world's fastest supercomputer to-date. New strategies are thus needed to expose the parallelism in legacy applications, while being amenable to efficient mapping to the underlying architecture. In this paper we discuss our experiences and strategies to port a scientific application, DCA++, to Summit. DCA++ is a high-performance research application that solves quantum many-body problems with a cutting edge quantum cluster algorithm, the dynamical cluster approximation. Our strategies aim to synergize the strengths of the different programming models in the code. These include: A) streamlining the interactions between the CPU threads and the GPUs, b) implementing computing kernels on the GPUs and decreasing CPU-GPU memory transfers, c) allowing asynchronous GPU communications, and d) increasing compute intensity by combining linear algebraic operations. Full-scale production runs using all 4600 Summit nodes attained a peak performance of 73.5 PFLOPS with a mixed precision implementation. We observed a perfect strong and weak scaling for the quantum Monte Carlo solver in DCA++, while encountering about 2x input/output (I/O) and MPI communication overhead on the time-To-solution for the full machine run. Our hardware agnostic optimizations are designed to alleviate the communication and I/O challenges observed, while improving the compute intensity and obtaining optimal performance on a complex, hybrid architecture like Summit.

AB - Optimizing scientific applications on today's accelerator-based high performance computing systems can be challenging, especially when multiple GPUs and CPUs with heterogeneous memories and persistent non-volatile memories are present. An example is Summit, an accelerator-based system at the Oak Ridge Leadership Computing Facility (OLCF) that is rated as the world's fastest supercomputer to-date. New strategies are thus needed to expose the parallelism in legacy applications, while being amenable to efficient mapping to the underlying architecture. In this paper we discuss our experiences and strategies to port a scientific application, DCA++, to Summit. DCA++ is a high-performance research application that solves quantum many-body problems with a cutting edge quantum cluster algorithm, the dynamical cluster approximation. Our strategies aim to synergize the strengths of the different programming models in the code. These include: A) streamlining the interactions between the CPU threads and the GPUs, b) implementing computing kernels on the GPUs and decreasing CPU-GPU memory transfers, c) allowing asynchronous GPU communications, and d) increasing compute intensity by combining linear algebraic operations. Full-scale production runs using all 4600 Summit nodes attained a peak performance of 73.5 PFLOPS with a mixed precision implementation. We observed a perfect strong and weak scaling for the quantum Monte Carlo solver in DCA++, while encountering about 2x input/output (I/O) and MPI communication overhead on the time-To-solution for the full machine run. Our hardware agnostic optimizations are designed to alleviate the communication and I/O challenges observed, while improving the compute intensity and obtaining optimal performance on a complex, hybrid architecture like Summit.

KW - CUDA

KW - CUDA aware MPI

KW - DCA

KW - QMC

KW - Quantum Monte Carlo

KW - Spectrum MPI

KW - Summit@OLCF

UR - http://www.scopus.com/inward/record.url?scp=85075444681&partnerID=8YFLogxK

U2 - 10.1109/PACT.2019.00041

DO - 10.1109/PACT.2019.00041

M3 - Conference contribution

AN - SCOPUS:85075444681

T3 - Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT

SP - 432

EP - 443

BT - Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019

Y2 - 21 September 2019 through 25 September 2019

ER -

Balduzzi G, Chatterjee A, Li YW, Doak PW, Haehner U, D'Azevedo EF et al. Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer. In Proceedings - 2019 28th International Conference on Parallel Architectures and Compilation Techniques, PACT 2019. Institute of Electrical and Electronics Engineers Inc. 2019. p. 432-443. 8891643. (Parallel Architectures and Compilation Techniques - Conference Proceedings, PACT). doi: 10.1109/PACT.2019.00041

Accelerating dca++ (dynamical cluster approximation) scientific application on the summit supercomputer

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this