HiCOO: Hierarchical cooperation for scalable communication in Global Address Space programming models on Cray XT systems

Weikuan Yu, Xinyu Que, Vinod Tipparaju, Jeffrey S. Vetter

Research output: Contribution to journalArticlepeer-review

Abstract

Global Address Space (GAS) programming models enable a convenient, shared-memory style addressing model. Typically this is realized through one-sided operations that can enable asynchronous communication and data movement. With the size of petascale systems reaching 10,000s of nodes and 100,000s of cores, the underlying runtime systems face critical challenges in (1) scalably managing resources (such as memory for communication buffers), and (2) gracefully handling unpredictable communication patterns and any associated contention. For any solution that addresses these resource scalability challenges, equally important is the need to maintain the performance of GAS programming models. In this paper, we describe a Hierarchical COOperation (HiCOO) architecture for scalable communication in GAS programming models. HiCOO formulates a cooperative communication architecture: with inter-node cooperation amongst multiple nodes (a.k.a multinode) and hierarchical cooperation among multinodes that are arranged in various virtual topologies. We have implemented HiCOO for a popular GAS runtime library, Aggregate Remote Memory Copy Interface (ARMCI). By extensively evaluating different virtual topologies in HiCOO in terms of their impact to memory scalability, network contention, and application performance, we identify MFCG as the most suitable virtual topology. The resulting HiCOO architecture is able to realize scalable resource management and achieve resilience to network contention, while at the same time maintaining or enhancing the performance of scientific applications. In one case, it reduces the total execution time of an NWChem application by 52%.

Original languageEnglish
Pages (from-to)1481-1492
Number of pages12
JournalJournal of Parallel and Distributed Computing
Volume72
Issue number11
DOIs
StatePublished - Nov 2012

Funding

This work was funded in part by NSF award CNS-1059376 , UT-Battelle grant ( UT-B-4000087151 ), and National Center for Computational Sciences . This research used resources of the National Center for Computational Sciences at Oak Ridge National Laboratory, which is supported by the Office of Science of the US Department of Energy under Contract No. DE-AC05-00OR22725 .

FundersFunder number
National Center for Computational Sciences
National Science Foundation1059376, CNS-1059376
U.S. Department of EnergyDE-AC05-00OR22725
Office of Science
UT-BattelleUT-B-4000087151

    Keywords

    • ARMCI
    • Contention
    • GAS
    • Multicore
    • Multinode
    • Virtual Topology

    Fingerprint

    Dive into the research topics of 'HiCOO: Hierarchical cooperation for scalable communication in Global Address Space programming models on Cray XT systems'. Together they form a unique fingerprint.

    Cite this