Storage-aware task scheduling for performance optimization of big data workflows

Qianwen Ye, Chase Q. Wu, Huiyan Cao, Nageswara S.V. Rao, Aiqin Hou

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

3 Scopus citations

Abstract

Many large-scale applications in various domains are generating big data, which are increasingly processed and analyzed by MapReduce-based workflows deployed in Hadoop systems. In addition to computing time, the makespan of such data-intensive workflows is also largely affected by communication cost. Particularly, there are two levels of data movement during the execution of distributed workflows in Hadoop: i) from map tasks to reduce tasks within each individual MapReduce module and ii) between each pair of adjacent modules in the workflow. Traditionally, these two aspects of network traffic have been treated separately as data locality at the task and module or job level, respectively. However, the interactions between these two levels of data movement may create complicated dynamics and their compound effects remain largely unexplored. In this paper, we formulate a task scheduling problem that considers data movement at both levels to minimize the end-to-end delay of a MapReduce-based workflow. We show this problem to be NP-complete, and design a storage-aware big data workflow scheduling algorithm, referred to as SA-BWS, to optimize workflow performance in Hadoop environments. The performance superiority of SA-BWS is illustrated by extensive simulations in comparison with the default workflow engine in Hadoop and existing scheduling methods.

Original languageEnglish
Title of host publicationProceedings - 16th IEEE International Symposium on Parallel and Distributed Processing with Applications, 17th IEEE International Conference on Ubiquitous Computing and Communications, 8th IEEE International Conference on Big Data and Cloud Computing, 11th IEEE International Conference on Social Computing and Networking and 8th IEEE International Conference on Sustainable Computing and Communications, ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018
EditorsJinjun Chen, Laurence T. Yang
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1095-1102
Number of pages8
ISBN (Electronic)9781728111414
DOIs
StatePublished - Jul 2 2018
Event16th IEEE International Symposium on Parallel and Distributed Processing with Applications, 17th IEEE International Conference on Ubiquitous Computing and Communications, 8th IEEE International Conference on Big Data and Cloud Computing, 11th IEEE International Conference on Social Computing and Networking and 8th IEEE International Conference on Sustainable Computing and Communications, ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018 - Melbourne, Australia
Duration: Dec 11 2018Dec 13 2018

Publication series

NameProceedings - 16th IEEE International Symposium on Parallel and Distributed Processing with Applications, 17th IEEE International Conference on Ubiquitous Computing and Communications, 8th IEEE International Conference on Big Data and Cloud Computing, 11th IEEE International Conference on Social Computing and Networking and 8th IEEE International Conference on Sustainable Computing and Communications, ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018

Conference

Conference16th IEEE International Symposium on Parallel and Distributed Processing with Applications, 17th IEEE International Conference on Ubiquitous Computing and Communications, 8th IEEE International Conference on Big Data and Cloud Computing, 11th IEEE International Conference on Social Computing and Networking and 8th IEEE International Conference on Sustainable Computing and Communications, ISPA/IUCC/BDCloud/SocialCom/SustainCom 2018
Country/TerritoryAustralia
CityMelbourne
Period12/11/1812/13/18

Funding

ACKNOWLEDGMENTS This research is sponsored by U.S. National Science Foundation under Grant No. CNS-1560698 with New Jersey Institute of Technology, and the Ministry of Science and Technology of China under Grant No. 2017YFB1300301 and National Nature Science Foundation of China under Grant No. 61472320 and U1609202 with Northwest University, P.R. China. REFERENCES

FundersFunder number
Ministry of Science and Technology of China2017YFB1300301
National Nature Science Foundation of China61472320, U1609202
New Jersey Institute of Technology
U.S. National Science FoundationCNS-1560698
Northwest University

    Keywords

    • Big data workflow
    • Data locality
    • MapReduce
    • Workflow optimization
    • Workflow scheduling

    Fingerprint

    Dive into the research topics of 'Storage-aware task scheduling for performance optimization of big data workflows'. Together they form a unique fingerprint.

    Cite this