Cross-geography scientific data transferring trends and behavior

Zhengchun Liu; Ian Foster; Rajkumar Kettimuthu; Nageswara S.V. Rao

doi:10.1145/3208040.3208053

Cross-geography scientific data transferring trends and behavior

Zhengchun Liu
, Ian Foster
, Rajkumar Kettimuthu
, Nageswara S.V. Rao

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

27 Scopus citations

Abstract

Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

Original language	English
Title of host publication	HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing
Publisher	Association for Computing Machinery, Inc
Pages	267-278
Number of pages	12
ISBN (Electronic)	9781450357852
DOIs	https://doi.org/10.1145/3208040.3208053
State	Published - Jun 11 2018
Event	27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018 - Tempe, United States Duration: Jun 11 2018 → Jun 15 2018

Publication series

Name	HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing

Conference

Conference	27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018
Country/Territory	United States
City	Tempe
Period	06/11/18 → 06/15/18

Funding

This material is based upon work supported by the U.S. Department of Energy, Office of Science, under contract number DE-AC02-06CH11357. We gratefully acknowledge the computing resources provided and operated by the Joint Laboratory for System Evaluation (JLSE) at Argonne National Laboratory. We also would like to thank the five anonymous reviewers for their helpful comments.

Keywords

File transfer
GridFTP
Usage management
Wide area network

Access to Document

10.1145/3208040.3208053

Cite this

Liu, Z., Foster, I., Kettimuthu, R., & Rao, N. S. V. (2018). Cross-geography scientific data transferring trends and behavior. In HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing (pp. 267-278). (HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing). Association for Computing Machinery, Inc. https://doi.org/10.1145/3208040.3208053

Liu, Zhengchun ; Foster, Ian ; Kettimuthu, Rajkumar et al. / Cross-geography scientific data transferring trends and behavior. HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc, 2018. pp. 267-278 (HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing).

@inproceedings{f84b4d8c22254f45bc22441f864d0112,

title = "Cross-geography scientific data transferring trends and behavior",

abstract = "Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.",

keywords = "File transfer, GridFTP, Usage management, Wide area network",

author = "Zhengchun Liu and Ian Foster and Rajkumar Kettimuthu and Rao, \{Nageswara S.V.\}",

note = "Publisher Copyright: {\textcopyright} 2018 Copyright held by the owner/author(s).; 27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018 ; Conference date: 11-06-2018 Through 15-06-2018",

year = "2018",

month = jun,

day = "11",

doi = "10.1145/3208040.3208053",

language = "English",

series = "HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing",

publisher = "Association for Computing Machinery, Inc",

pages = "267--278",

booktitle = "HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing",

}

Liu, Z, Foster, I, Kettimuthu, R & Rao, NSV 2018, Cross-geography scientific data transferring trends and behavior. in HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing. HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing, Association for Computing Machinery, Inc, pp. 267-278, 27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018, Tempe, United States, 06/11/18. https://doi.org/10.1145/3208040.3208053

Cross-geography scientific data transferring trends and behavior. / Liu, Zhengchun; Foster, Ian; Kettimuthu, Rajkumar et al.
HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc, 2018. p. 267-278 (HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Cross-geography scientific data transferring trends and behavior

AU - Liu, Zhengchun

AU - Foster, Ian

AU - Kettimuthu, Rajkumar

AU - Rao, Nageswara S.V.

PY - 2018/6/11

Y1 - 2018/6/11

N2 - Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

AB - Wide area data transfers play an important role in many science applications but rely on expensive infrastructure that often delivers disappointing performance in practice. In response, we present a systematic examination of a large set of data transfer log data to characterize transfer characteristics, including the nature of the datasets transferred, achieved throughput, user behavior, and resource usage. This analysis yields new insights that can help design better data transfer tools, optimize networking and edge resources used for transfers, and improve the performance and experience for end users. Our analysis shows that (i) most of the datasets as well as individual files transferred are very small; (ii) data corruption is not negligible for large data transfers; and (iii) the data transfer nodes utilization is low. Insights gained from our analysis suggest directions for further analysis.

KW - File transfer

KW - GridFTP

KW - Usage management

KW - Wide area network

UR - https://www.scopus.com/pages/publications/85050097894

U2 - 10.1145/3208040.3208053

DO - 10.1145/3208040.3208053

M3 - Conference contribution

AN - SCOPUS:85050097894

T3 - HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing

SP - 267

EP - 278

BT - HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing

PB - Association for Computing Machinery, Inc

T2 - 27th ACM International Symposium on High-Performance Parallel and Distributed Computing, HPDC 2018

Y2 - 11 June 2018 through 15 June 2018

ER -

Liu Z, Foster I, Kettimuthu R, Rao NSV. Cross-geography scientific data transferring trends and behavior. In HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing. Association for Computing Machinery, Inc. 2018. p. 267-278. (HPDC 2018 - Proceedings of the 2018 International Symposium on High-Performance Parallel and Distributed Computing). doi: 10.1145/3208040.3208053

Cross-geography scientific data transferring trends and behavior

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this