Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all

Ana Gainaru; Richard L. Graham; Artem Polyakov; Gilad Shainer

doi:10.1145/2966884.2966918

Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all

Ana Gainaru, Richard L. Graham, Artem Polyakov, Gilad Shainer

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

10 Scopus citations

Abstract

The MPI all-to-all algorithm is a data intensive, high-cost collective algorithm used by many scientific High Performance Computing applications. Optimizations for small data exchange use aggregation techniques, such as the Bruck algorithm, to minimize the number of messages sent, and minimize overall operation latency. This paper presents three variants of the Bruck algorithm, which differ in the way data is laid out in memory at intermediate steps of the algorithm. Mellanox's InfiniBand support for Host Channel Adapter (HCA) hardware scatter/gather is used selectively to replace CPU-based buffer packing and unpacking. Using this offload capability reduces the eight and sixteen byte all-to-all latency on 1024 MPI Processes by 9.7% and 9.1%, respectively. The optimization accounts for a decrease in the total memory handling time of 40.6% and 57.9%, respectively.

Original language	English
Title of host publication	Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016
Publisher	Association for Computing Machinery
Pages	167-179
Number of pages	13
ISBN (Electronic)	9781450342346
DOIs	https://doi.org/10.1145/2966884.2966918
State	Published - Sep 25 2016
Externally published	Yes
Event	23rd European MPI Users' Group Meeting, EuroMPI 2016 - Edinburgh, United Kingdom Duration: Sep 25 2016 → Sep 28 2016

Publication series

Name	ACM International Conference Proceeding Series
Volume	25-28-September-2016

Conference

Conference	23rd European MPI Users' Group Meeting, EuroMPI 2016
Country/Territory	United Kingdom
City	Edinburgh
Period	09/25/16 → 09/28/16

Keywords

All-to-all
Collective communication
MPI
Network offload

Access to Document

10.1145/2966884.2966918

Cite this

Gainaru, A., Graham, R. L., Polyakov, A., & Shainer, G. (2016). Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all. In Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016 (pp. 167-179). (ACM International Conference Proceeding Series; Vol. 25-28-September-2016). Association for Computing Machinery. https://doi.org/10.1145/2966884.2966918

@inproceedings{5b3cff2be8444acb90e3b8df68dffb5e,

title = "Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all",

abstract = "The MPI all-to-all algorithm is a data intensive, high-cost collective algorithm used by many scientific High Performance Computing applications. Optimizations for small data exchange use aggregation techniques, such as the Bruck algorithm, to minimize the number of messages sent, and minimize overall operation latency. This paper presents three variants of the Bruck algorithm, which differ in the way data is laid out in memory at intermediate steps of the algorithm. Mellanox's InfiniBand support for Host Channel Adapter (HCA) hardware scatter/gather is used selectively to replace CPU-based buffer packing and unpacking. Using this offload capability reduces the eight and sixteen byte all-to-all latency on 1024 MPI Processes by 9.7% and 9.1%, respectively. The optimization accounts for a decrease in the total memory handling time of 40.6% and 57.9%, respectively.",

keywords = "All-to-all, Collective communication, MPI, Network offload",

author = "Ana Gainaru and Graham, {Richard L.} and Artem Polyakov and Gilad Shainer",

note = "Publisher Copyright: {\textcopyright} 2016 ACM.; 23rd European MPI Users' Group Meeting, EuroMPI 2016 ; Conference date: 25-09-2016 Through 28-09-2016",

year = "2016",

month = sep,

day = "25",

doi = "10.1145/2966884.2966918",

language = "English",

series = "ACM International Conference Proceeding Series",

publisher = "Association for Computing Machinery",

pages = "167--179",

booktitle = "Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016",

}

Gainaru, A, Graham, RL, Polyakov, A & Shainer, G 2016, Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all. in Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016. ACM International Conference Proceeding Series, vol. 25-28-September-2016, Association for Computing Machinery, pp. 167-179, 23rd European MPI Users' Group Meeting, EuroMPI 2016, Edinburgh, United Kingdom, 09/25/16. https://doi.org/10.1145/2966884.2966918

Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all. / Gainaru, Ana; Graham, Richard L.; Polyakov, Artem et al.
Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016. Association for Computing Machinery, 2016. p. 167-179 (ACM International Conference Proceeding Series; Vol. 25-28-September-2016).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all

AU - Gainaru, Ana

AU - Graham, Richard L.

AU - Polyakov, Artem

AU - Shainer, Gilad

PY - 2016/9/25

Y1 - 2016/9/25

N2 - The MPI all-to-all algorithm is a data intensive, high-cost collective algorithm used by many scientific High Performance Computing applications. Optimizations for small data exchange use aggregation techniques, such as the Bruck algorithm, to minimize the number of messages sent, and minimize overall operation latency. This paper presents three variants of the Bruck algorithm, which differ in the way data is laid out in memory at intermediate steps of the algorithm. Mellanox's InfiniBand support for Host Channel Adapter (HCA) hardware scatter/gather is used selectively to replace CPU-based buffer packing and unpacking. Using this offload capability reduces the eight and sixteen byte all-to-all latency on 1024 MPI Processes by 9.7% and 9.1%, respectively. The optimization accounts for a decrease in the total memory handling time of 40.6% and 57.9%, respectively.

AB - The MPI all-to-all algorithm is a data intensive, high-cost collective algorithm used by many scientific High Performance Computing applications. Optimizations for small data exchange use aggregation techniques, such as the Bruck algorithm, to minimize the number of messages sent, and minimize overall operation latency. This paper presents three variants of the Bruck algorithm, which differ in the way data is laid out in memory at intermediate steps of the algorithm. Mellanox's InfiniBand support for Host Channel Adapter (HCA) hardware scatter/gather is used selectively to replace CPU-based buffer packing and unpacking. Using this offload capability reduces the eight and sixteen byte all-to-all latency on 1024 MPI Processes by 9.7% and 9.1%, respectively. The optimization accounts for a decrease in the total memory handling time of 40.6% and 57.9%, respectively.

KW - All-to-all

KW - Collective communication

KW - MPI

KW - Network offload

UR - http://www.scopus.com/inward/record.url?scp=84995662375&partnerID=8YFLogxK

U2 - 10.1145/2966884.2966918

DO - 10.1145/2966884.2966918

M3 - Conference contribution

AN - SCOPUS:84995662375

T3 - ACM International Conference Proceeding Series

SP - 167

EP - 179

BT - Proceedings of the 23rd European MPI Users' Group Meeting, EuroMPI 2016

PB - Association for Computing Machinery

T2 - 23rd European MPI Users' Group Meeting, EuroMPI 2016

Y2 - 25 September 2016 through 28 September 2016

ER -

Using infiniband hardware gather-scatter capabilities to optimize MPI all-to-all

Abstract

Publication series

Conference

Keywords

Access to Document

Other files and links

Fingerprint

Cite this