Throughput Analytics of Data Transfer Infrastructures

Nageswara S.V. Rao; Qiang Liu; Zhengchun Liu; Rajkumar Kettimuthu; Ian Foster

doi:10.1007/978-3-030-12971-2_2

Throughput Analytics of Data Transfer Infrastructures

Nageswara S.V. Rao, Qiang Liu, Zhengchun Liu, Rajkumar Kettimuthu, Ian Foster

Autonomous and Complex Systems

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.

Original language	English
Title of host publication	Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings
Editors	Honghao Gao, Huaikou Miao, Xiaoxian Yang, Yuyu Yin
Publisher	Springer Verlag
Pages	20-40
Number of pages	21
ISBN (Print)	9783030129705
DOIs	https://doi.org/10.1007/978-3-030-12971-2_2
State	Published - 2019
Event	13th EAI International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TridentCom 2018 - Shanghai, China Duration: Dec 1 2018 → Dec 3 2018

Publication series

Name	Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST
Volume	270
ISSN (Print)	1867-8211

Conference

Conference	13th EAI International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TridentCom 2018
Country/Territory	China
City	Shanghai
Period	12/1/18 → 12/3/18

Funding

This work is funded by RAMSES project and the Applied Mathematics Program, Office of Advanced Computing Research, U.S. Department of Energy, and by Extreme Scale Systems Center, sponsored by U. S. Department of Defense, and performed at Oak Ridge National Laboratory managed by UT-Battelle, LLC for U.S. Department of Energy under Contract No. DE-AC05-00OR22725.

Keywords

Data transfer
Infrastructure
Throughput profile

Access to Document

10.1007/978-3-030-12971-2_2

Cite this

Rao, N. S. V., Liu, Q., Liu, Z., Kettimuthu, R., & Foster, I. (2019). Throughput Analytics of Data Transfer Infrastructures. In H. Gao, H. Miao, X. Yang, & Y. Yin (Eds.), Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings (pp. 20-40). (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 270). Springer Verlag. https://doi.org/10.1007/978-3-030-12971-2_2

Rao, Nageswara S.V. ; Liu, Qiang ; Liu, Zhengchun et al. / Throughput Analytics of Data Transfer Infrastructures. Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings. editor / Honghao Gao ; Huaikou Miao ; Xiaoxian Yang ; Yuyu Yin. Springer Verlag, 2019. pp. 20-40 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST).

@inproceedings{b0d1e0dda6484a799ac2316e25748ade,

title = "Throughput Analytics of Data Transfer Infrastructures",

abstract = "To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.",

keywords = "Data transfer, Infrastructure, Throughput profile",

author = "Rao, {Nageswara S.V.} and Qiang Liu and Zhengchun Liu and Rajkumar Kettimuthu and Ian Foster",

note = "Publisher Copyright: {\textcopyright} 2019, ICST Institute for Computer Sciences, Social Informatics and Telecommunications Engineering.; 13th EAI International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TridentCom 2018 ; Conference date: 01-12-2018 Through 03-12-2018",

year = "2019",

doi = "10.1007/978-3-030-12971-2_2",

language = "English",

isbn = "9783030129705",

series = "Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST",

publisher = "Springer Verlag",

pages = "20--40",

editor = "Honghao Gao and Huaikou Miao and Xiaoxian Yang and Yuyu Yin",

booktitle = "Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings",

}

Rao, NSV, Liu, Q, Liu, Z, Kettimuthu, R & Foster, I 2019, Throughput Analytics of Data Transfer Infrastructures. in H Gao, H Miao, X Yang & Y Yin (eds), Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings. Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST, vol. 270, Springer Verlag, pp. 20-40, 13th EAI International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TridentCom 2018, Shanghai, China, 12/1/18. https://doi.org/10.1007/978-3-030-12971-2_2

Throughput Analytics of Data Transfer Infrastructures. / Rao, Nageswara S.V.; Liu, Qiang; Liu, Zhengchun et al.
Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings. ed. / Honghao Gao; Huaikou Miao; Xiaoxian Yang; Yuyu Yin. Springer Verlag, 2019. p. 20-40 (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST; Vol. 270).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Throughput Analytics of Data Transfer Infrastructures

AU - Rao, Nageswara S.V.

AU - Liu, Qiang

AU - Liu, Zhengchun

AU - Kettimuthu, Rajkumar

AU - Foster, Ian

PY - 2019

Y1 - 2019

N2 - To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.

AB - To support increasingly distributed scientific and big-data applications, powerful data transfer infrastructures are being built with dedicated networks and software frameworks customized to distributed file systems and data transfer nodes. The data transfer performance of such infrastructures critically depends on the combined choices of file, disk, and host systems as well as network protocols and file transfer software, all of which may vary across sites. The randomness of throughput measurements makes it challenging to assess the impact of these choices on the performance of infrastructure or its parts. We propose regression-based throughput profiles by aggregating measurements from sites of the infrastructure, with RTT as the independent variable. The peak values and convex-concave shape of a profile together determine the overall throughput performance of memory and file transfers, and its variations show the performance differences among the sites. We then present projection and difference operators, and coefficients of throughput profiles to characterize the performance of infrastructure and its parts, including sites and file transfer tools. In particular, the utilization-concavity coefficient provides a value in the range [0, 1] that reflects overall transfer effectiveness. We present results of measurements collected using (i) testbed experiments over dedicated 0–366 ms 10 Gbps connections with combinations of TCP versions, file systems, host systems and transfer tools, and (ii) Globus GridFTP transfers over production infrastructure with varying site configurations.

KW - Data transfer

KW - Infrastructure

KW - Throughput profile

UR - http://www.scopus.com/inward/record.url?scp=85063038780&partnerID=8YFLogxK

U2 - 10.1007/978-3-030-12971-2_2

DO - 10.1007/978-3-030-12971-2_2

M3 - Conference contribution

AN - SCOPUS:85063038780

SN - 9783030129705

T3 - Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST

SP - 20

EP - 40

BT - Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings

A2 - Gao, Honghao

A2 - Miao, Huaikou

A2 - Yang, Xiaoxian

A2 - Yin, Yuyu

PB - Springer Verlag

T2 - 13th EAI International Conference on Testbeds and Research Infrastructures for the Development of Networks and Communities, TridentCom 2018

Y2 - 1 December 2018 through 3 December 2018

ER -

Rao NSV, Liu Q, Liu Z, Kettimuthu R, Foster I. Throughput Analytics of Data Transfer Infrastructures. In Gao H, Miao H, Yang X, Yin Y, editors, Testbeds and Research Infrastructures for the Development of Networks and Communities - 13th EAI International Conference, TridentCom 2018, Proceedings. Springer Verlag. 2019. p. 20-40. (Lecture Notes of the Institute for Computer Sciences, Social-Informatics and Telecommunications Engineering, LNICST). doi: 10.1007/978-3-030-12971-2_2

Throughput Analytics of Data Transfer Infrastructures

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this