Optimizing network performance of computing pipelines in distributed environments

Qishi Wu, Yi Gu, Mengxia Zhu, Nageswara S.V. Rao

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

20 Scopus citations

Abstract

Supporting high performance computing pipelines over wide-area networks is critical to enabling large-scale distributed scientific applications that require fast responses for interactive operations or smooth flows for data streaming. We construct analytical cost models for computing modules, network nodes, and communication links to estimate the computing times on nodes and the data transport times over connections. Based on these time estimates, we present the Efficient Linear Pipeline Configuration method based on dynamic programming that partitions the pipeline modules into groups and strategically maps them onto a set of selected computing nodes in a network to achieve minimum end-to-end delay or maximum frame rate. We implemented this method and evaluated its effectiveness with experiments on a large set of simulated application pipelines and computing networks. The experimental results show that the proposed method outperforms the Streamline and Greedy algorithms. These results, together with polynomial computational complexity, make our method a potential scalable solution for large practical deployments.

Original languageEnglish
Title of host publicationIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM
DOIs
StatePublished - 2008
EventIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium - Miami, FL, United States
Duration: Apr 14 2008Apr 18 2008

Publication series

NameIPDPS Miami 2008 - Proceedings of the 22nd IEEE International Parallel and Distributed Processing Symposium, Program and CD-ROM

Conference

ConferenceIPDPS 2008 - 22nd IEEE International Parallel and Distributed Processing Symposium
Country/TerritoryUnited States
CityMiami, FL
Period04/14/0804/18/08

Fingerprint

Dive into the research topics of 'Optimizing network performance of computing pipelines in distributed environments'. Together they form a unique fingerprint.

Cite this