A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems

Awais Khan; Chang Gyu Lee; Prince Hamandawana; Sungyong Park; Youngjae Kim

doi:10.1109/MASCOTS.2018.00016

A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems

Awais Khan
, Chang Gyu Lee
, Prince Hamandawana
, Sungyong Park
, Youngjae Kim

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

21 Scopus citations

Abstract

Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-Tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared-nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.

Original language	English
Title of host publication	Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	87-93
Number of pages	7
ISBN (Electronic)	9781538668863
DOIs	https://doi.org/10.1109/MASCOTS.2018.00016
State	Published - Nov 7 2018
Externally published	Yes
Event	26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018 - Milwaukee, United States Duration: Sep 25 2018 → Sep 28 2018

Publication series

Name	Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018

Conference

Conference	26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018
Country/Territory	United States
City	Milwaukee
Period	09/25/18 → 09/28/18

Funding

This work was supported by Institute for Information & communications TechnologyPromotion(IITP) grant funded by the Korea government(MSIT) (No.2014-0-00035).

Keywords

Data Deduplication
Distributed Storage Systems
Distributed and cloud computing
Storage and file systems

Access to Document

10.1109/MASCOTS.2018.00016

Cite this

Khan, A., Lee, C. G., Hamandawana, P., Park, S., & Kim, Y. (2018). A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems. In Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018 (pp. 87-93). Article 8526874 (Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/MASCOTS.2018.00016

Khan, Awais ; Lee, Chang Gyu ; Hamandawana, Prince et al. / A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems. Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. pp. 87-93 (Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018).

@inproceedings{954572b825894eebb97c9837e53e4a84,

title = "A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems",

abstract = "Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-Tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared-nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.",

keywords = "Data Deduplication, Distributed Storage Systems, Distributed and cloud computing, Storage and file systems",

author = "Awais Khan and Lee, \{Chang Gyu\} and Prince Hamandawana and Sungyong Park and Youngjae Kim",

note = "Publisher Copyright: {\textcopyright} 2018 IEEE.; 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018 ; Conference date: 25-09-2018 Through 28-09-2018",

year = "2018",

month = nov,

day = "7",

doi = "10.1109/MASCOTS.2018.00016",

language = "English",

series = "Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "87--93",

booktitle = "Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018",

}

Khan, A, Lee, CG, Hamandawana, P, Park, S & Kim, Y 2018, A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems. in Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018., 8526874, Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018, Institute of Electrical and Electronics Engineers Inc., pp. 87-93, 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018, Milwaukee, United States, 09/25/18. https://doi.org/10.1109/MASCOTS.2018.00016

A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems. / Khan, Awais; Lee, Chang Gyu; Hamandawana, Prince et al.
Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018. Institute of Electrical and Electronics Engineers Inc., 2018. p. 87-93 8526874 (Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution › peer-review

TY - GEN

T1 - A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems

AU - Khan, Awais

AU - Lee, Chang Gyu

AU - Hamandawana, Prince

AU - Park, Sungyong

AU - Kim, Youngjae

PY - 2018/11/7

Y1 - 2018/11/7

N2 - Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-Tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared-nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.

AB - Deduplication has been largely employed in distributed storage systems to improve space efficiency. Traditional deduplication research ignores the design specifications of shared-nothing distributed storage systems such as no central metadata bottleneck, scalability, and storage rebalancing. Further, deduplication introduces transactional changes, which are prone to errors in the event of a system failure, resulting in inconsistencies in data and deduplication metadata. In this paper, we propose a robust, fault-Tolerant and scalable cluster-wide deduplication that can eliminate duplicate copies across the cluster. We design a distributed deduplication metadata shard which guarantees performance scalability while preserving the design constraints of shared-nothing storage systems. The placement of chunks and deduplication metadata is made cluster-wide based on the content fingerprint of chunks. To ensure transactional consistency and garbage identification, we employ a flag-based asynchronous consistency mechanism. We implement the proposed deduplication on Ceph. The evaluation shows high disk-space savings with minimal performance degradation as well as high robustness in the event of sudden server failure.

KW - Data Deduplication

KW - Distributed Storage Systems

KW - Distributed and cloud computing

KW - Storage and file systems

UR - https://www.scopus.com/pages/publications/85058318460

U2 - 10.1109/MASCOTS.2018.00016

DO - 10.1109/MASCOTS.2018.00016

M3 - Conference contribution

AN - SCOPUS:85058318460

T3 - Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018

SP - 87

EP - 93

BT - Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018

Y2 - 25 September 2018 through 28 September 2018

ER -

Khan A, Lee CG, Hamandawana P, Park S, Kim Y. A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems. In Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018. Institute of Electrical and Electronics Engineers Inc. 2018. p. 87-93. 8526874. (Proceedings - 26th IEEE International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems, MASCOTS 2018). doi: 10.1109/MASCOTS.2018.00016

A robust fault-tolerant and scalable cluster-wide deduplication for shared-nothing storage systems

Abstract

Publication series

Conference

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this