Building a wide-area file transfer performance predictor: An empirical study

Zhengchun Liu; Rajkumar Kettimuthu; Prasanna Balaprakash; Nageswara S.V. Rao; Ian Foster

doi:10.1007/978-3-030-19945-6_5

Building a wide-area file transfer performance predictor: An empirical study

Zhengchun Liu, Rajkumar Kettimuthu, Prasanna Balaprakash, Nageswara S.V. Rao, Ian Foster

Autonomous and Complex Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

1 Scopus citations

Abstract

Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

Original language	English
Title of host publication	Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers
Editors	Paul Mühlethaler, Selma Boumerdassi, Éric Renault
Publisher	Springer Verlag
Pages	56-78
Number of pages	23
ISBN (Print)	9783030199449
DOIs	https://doi.org/10.1007/978-3-030-19945-6_5
State	Published - 2019
Event	1st International Conference on Machine Learning for Networking, MLN 2018 - Paris, France Duration: Nov 27 2018 → Nov 29 2018

Publication series

Name	Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume	11407 LNCS
ISSN (Print)	0302-9743
ISSN (Electronic)	1611-3349

Conference

Conference	1st International Conference on Machine Learning for Networking, MLN 2018
Country/Territory	France
City	Paris
Period	11/27/18 → 11/29/18

Funding

Acknowledgments. This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contract DE-AC02-06CH11357. We gratefully acknowledge the computing resources provided and operated by the Joint Laboratory for System Evaluation (JLSE) at Argonne National Laboratory.

Access to Document

10.1007/978-3-030-19945-6_5

Cite this

Liu, Z., Kettimuthu, R., Balaprakash, P., Rao, N. S. V., & Foster, I. (2019). Building a wide-area file transfer performance predictor: An empirical study. In P. Mühlethaler, S. Boumerdassi, & É. Renault (Eds.), Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers (pp. 56-78). (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11407 LNCS). Springer Verlag. https://doi.org/10.1007/978-3-030-19945-6_5

Liu, Zhengchun ; Kettimuthu, Rajkumar ; Balaprakash, Prasanna et al. / Building a wide-area file transfer performance predictor : An empirical study. Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers. editor / Paul Mühlethaler ; Selma Boumerdassi ; Éric Renault. Springer Verlag, 2019. pp. 56-78 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)).

@inproceedings{6850571b5b364a7b9c62de2b74887e1b,

title = "Building a wide-area file transfer performance predictor: An empirical study",

abstract = "Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.",

author = "Zhengchun Liu and Rajkumar Kettimuthu and Prasanna Balaprakash and Rao, {Nageswara S.V.} and Ian Foster",

note = "Publisher Copyright: {\textcopyright} Springer Nature Switzerland AG 2019.; 1st International Conference on Machine Learning for Networking, MLN 2018 ; Conference date: 27-11-2018 Through 29-11-2018",

year = "2019",

doi = "10.1007/978-3-030-19945-6_5",

language = "English",

isbn = "9783030199449",

series = "Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)",

publisher = "Springer Verlag",

pages = "56--78",

editor = "Paul M{\"u}hlethaler and Selma Boumerdassi and {\'E}ric Renault",

booktitle = "Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers",

}

Liu, Z, Kettimuthu, R, Balaprakash, P , Rao, NSV & Foster, I 2019, Building a wide-area file transfer performance predictor: An empirical study. in P Mühlethaler, S Boumerdassi & É Renault (eds), Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), vol. 11407 LNCS, Springer Verlag, pp. 56-78, 1st International Conference on Machine Learning for Networking, MLN 2018, Paris, France, 11/27/18. https://doi.org/10.1007/978-3-030-19945-6_5

Building a wide-area file transfer performance predictor: An empirical study. / Liu, Zhengchun; Kettimuthu, Rajkumar; Balaprakash, Prasanna et al.
Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers. ed. / Paul Mühlethaler; Selma Boumerdassi; Éric Renault. Springer Verlag, 2019. p. 56-78 (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics); Vol. 11407 LNCS).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Building a wide-area file transfer performance predictor

T2 - 1st International Conference on Machine Learning for Networking, MLN 2018

AU - Liu, Zhengchun

AU - Kettimuthu, Rajkumar

AU - Balaprakash, Prasanna

AU - Rao, Nageswara S.V.

AU - Foster, Ian

PY - 2019

Y1 - 2019

N2 - Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

AB - Wide-area data transfer is central to geographically distributed scientific workflows. Faster delivery of data is important for these workflows. Predictability is equally (or even more) important. With the goal of providing a reasonably accurate estimate of data transfer time to improve resource allocation & scheduling for workflows and enable end-to-end data transfer optimization, we apply machine learning methods to develop predictive models for data transfer times over a variety of wide area networks. To build and evaluate these models, we use 201,388 transfers, involving 759 million files totaling 9 PB transferred, over 115 heavily used source-destination pairs (“edges”) between 135 unique endpoints. We evaluate the models for different retraining frequencies and different window size of history data. In the best case, the resulting models have a median prediction error of ≤21% for 50% of the edges, and ≤32% for 75% of the edges. We present a detailed analysis of these results that provides insights into the cause of some of the high errors. We envision that the performance predictor will be informative for scheduling geo-distributed workflows. The insights also suggest obvious directions for both further analysis and transfer service optimization.

UR - http://www.scopus.com/inward/record.url?scp=85065829273&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-19945-6_5

DO - 10.1007/978-3-030-19945-6_5

M3 - Conference contribution

AN - SCOPUS:85065829273

SN - 9783030199449

T3 - Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

SP - 56

EP - 78

BT - Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers

A2 - Mühlethaler, Paul

A2 - Boumerdassi, Selma

A2 - Renault, Éric

PB - Springer Verlag

Y2 - 27 November 2018 through 29 November 2018

ER -

Liu Z, Kettimuthu R, Balaprakash P , Rao NSV, Foster I. Building a wide-area file transfer performance predictor: An empirical study. In Mühlethaler P, Boumerdassi S, Renault É, editors, Machine Learning for Networking - 1st International Conference, MLN 2018, Revised Selected Papers. Springer Verlag. 2019. p. 56-78. (Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)). doi: 10.1007/978-3-030-19945-6_5

Building a wide-area file transfer performance predictor: An empirical study

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this