Design and analysis of fault tolerance mechanisms for big data transfers

Preethika Kasu; Youngjae Kim; Sungyong Park; Scott Atchley; Geoffroy R. Vallée

doi:10.1109/CLUSTER.2016.74

Design and analysis of fault tolerance mechanisms for big data transfers

Preethika Kasu, Youngjae Kim, Sungyong Park, Scott Atchley, Geoffroy R. Vallée

National Center for Computational Scienc

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

3 Scopus citations

Abstract

Increased growth of the data and the need to move the data between data centers, demands, an efficient data transfer tool which can, not only transfer the data at higher rates but also handle the faults occurred during the transfer. Absence of fault tolerance mechanisms, would need to retransmit the whole data, in case of any error during the transfer. Hence, fault tolerance is an important aspect of big data transfer tools. In this paper, we have considered LADS data transfer tool, which proved to be superior to the existing data transfer tools with respect to the speed of the transfer. However, absence of fault tolerance implementation in LADS might result in data retransmission and congestion issues upon errors. In this paper, we design and analyze fault tolerance mechanisms which can be used with LADS data transfer tool. We have proposed three different fault tolerance mechanisms, File logging, Transaction logging and Universal logging. Also, we have analyzed the space and performance overhead of these fault tolerance mechanisms on LADS data transfer tool.

Original language	English
Title of host publication	Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	138-139
Number of pages	2
ISBN (Electronic)	9781509036530
DOIs	https://doi.org/10.1109/CLUSTER.2016.74
State	Published - Dec 6 2016
Event	2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 - Taipei, Taiwan, Province of China Duration: Sep 13 2016 → Sep 15 2016

Publication series

Name	Proceedings - IEEE International Conference on Cluster Computing, ICCC
ISSN (Print)	1552-5244

Conference

Conference	2016 IEEE International Conference on Cluster Computing, CLUSTER 2016
Country/Territory	Taiwan, Province of China
City	Taipei
Period	09/13/16 → 09/15/16

Funding

This work was supported by the Institute for Information & communications Technology Promotion(IITP) grant (No. R0190-15-2012) and by the National Research Foundation of Korea (NRF) grant (No. 2015R1C1A1A0152105), both funded by the Korea Government (MISP).

Access to Document

10.1109/CLUSTER.2016.74

Cite this

Kasu, P., Kim, Y., Park, S., Atchley, S., & Vallée, G. R. (2016). Design and analysis of fault tolerance mechanisms for big data transfers. In Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 (pp. 138-139). Article 7776490 (Proceedings - IEEE International Conference on Cluster Computing, ICCC). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CLUSTER.2016.74

@inproceedings{53792d8873ec4aeeb29b1b0081a43b8f,

title = "Design and analysis of fault tolerance mechanisms for big data transfers",

abstract = "Increased growth of the data and the need to move the data between data centers, demands, an efficient data transfer tool which can, not only transfer the data at higher rates but also handle the faults occurred during the transfer. Absence of fault tolerance mechanisms, would need to retransmit the whole data, in case of any error during the transfer. Hence, fault tolerance is an important aspect of big data transfer tools. In this paper, we have considered LADS data transfer tool, which proved to be superior to the existing data transfer tools with respect to the speed of the transfer. However, absence of fault tolerance implementation in LADS might result in data retransmission and congestion issues upon errors. In this paper, we design and analyze fault tolerance mechanisms which can be used with LADS data transfer tool. We have proposed three different fault tolerance mechanisms, File logging, Transaction logging and Universal logging. Also, we have analyzed the space and performance overhead of these fault tolerance mechanisms on LADS data transfer tool.",

author = "Preethika Kasu and Youngjae Kim and Sungyong Park and Scott Atchley and Vall{\'e}e, {Geoffroy R.}",

note = "Publisher Copyright: {\textcopyright} 2016 IEEE.; 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016 ; Conference date: 13-09-2016 Through 15-09-2016",

year = "2016",

month = dec,

day = "6",

doi = "10.1109/CLUSTER.2016.74",

language = "English",

series = "Proceedings - IEEE International Conference on Cluster Computing, ICCC",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "138--139",

booktitle = "Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016",

}

Kasu, P, Kim, Y, Park, S, Atchley, S & Vallée, GR 2016, Design and analysis of fault tolerance mechanisms for big data transfers. in Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016., 7776490, Proceedings - IEEE International Conference on Cluster Computing, ICCC, Institute of Electrical and Electronics Engineers Inc., pp. 138-139, 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016, Taipei, Taiwan, Province of China, 09/13/16. https://doi.org/10.1109/CLUSTER.2016.74

Design and analysis of fault tolerance mechanisms for big data transfers. / Kasu, Preethika; Kim, Youngjae; Park, Sungyong et al.
Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016. Institute of Electrical and Electronics Engineers Inc., 2016. p. 138-139 7776490 (Proceedings - IEEE International Conference on Cluster Computing, ICCC).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - Design and analysis of fault tolerance mechanisms for big data transfers

AU - Kasu, Preethika

AU - Kim, Youngjae

AU - Park, Sungyong

AU - Atchley, Scott

AU - Vallée, Geoffroy R.

PY - 2016/12/6

Y1 - 2016/12/6

N2 - Increased growth of the data and the need to move the data between data centers, demands, an efficient data transfer tool which can, not only transfer the data at higher rates but also handle the faults occurred during the transfer. Absence of fault tolerance mechanisms, would need to retransmit the whole data, in case of any error during the transfer. Hence, fault tolerance is an important aspect of big data transfer tools. In this paper, we have considered LADS data transfer tool, which proved to be superior to the existing data transfer tools with respect to the speed of the transfer. However, absence of fault tolerance implementation in LADS might result in data retransmission and congestion issues upon errors. In this paper, we design and analyze fault tolerance mechanisms which can be used with LADS data transfer tool. We have proposed three different fault tolerance mechanisms, File logging, Transaction logging and Universal logging. Also, we have analyzed the space and performance overhead of these fault tolerance mechanisms on LADS data transfer tool.

AB - Increased growth of the data and the need to move the data between data centers, demands, an efficient data transfer tool which can, not only transfer the data at higher rates but also handle the faults occurred during the transfer. Absence of fault tolerance mechanisms, would need to retransmit the whole data, in case of any error during the transfer. Hence, fault tolerance is an important aspect of big data transfer tools. In this paper, we have considered LADS data transfer tool, which proved to be superior to the existing data transfer tools with respect to the speed of the transfer. However, absence of fault tolerance implementation in LADS might result in data retransmission and congestion issues upon errors. In this paper, we design and analyze fault tolerance mechanisms which can be used with LADS data transfer tool. We have proposed three different fault tolerance mechanisms, File logging, Transaction logging and Universal logging. Also, we have analyzed the space and performance overhead of these fault tolerance mechanisms on LADS data transfer tool.

UR - http://www.scopus.com/inward/record.url?scp=85013197163&partnerID=8YFLogxK

U2 - 10.1109/CLUSTER.2016.74

DO - 10.1109/CLUSTER.2016.74

M3 - Conference contribution

AN - SCOPUS:85013197163

T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC

SP - 138

EP - 139

BT - Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016

Y2 - 13 September 2016 through 15 September 2016

ER -

Kasu P, Kim Y, Park S, Atchley S, Vallée GR. Design and analysis of fault tolerance mechanisms for big data transfers. In Proceedings - 2016 IEEE International Conference on Cluster Computing, CLUSTER 2016. Institute of Electrical and Electronics Engineers Inc. 2016. p. 138-139. 7776490. (Proceedings - IEEE International Conference on Cluster Computing, ICCC). doi: 10.1109/CLUSTER.2016.74

Design and analysis of fault tolerance mechanisms for big data transfers

Abstract

Publication series

Conference

Funding

Access to Document

Other files and links

Fingerprint

Cite this