Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Qing Liu; Jeremy Logan; Yuan Tian; Hasan Abbasi; Norbert Podhorszki; Jong Youl Choi; Scott Klasky; Roselyne Tchoua; Jay Lofstead; Ron Oldfield; Manish Parashar; Nagiza Samatova; Karsten Schwan; Arie Shoshani; Matthew Wolf; Kesheng Wu; Weikuan Yu

doi:10.1002/cpe.3125

Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Qing Liu, Jeremy Logan, Yuan Tian, Hasan Abbasi, Norbert Podhorszki, Jong Youl Choi, Scott Klasky, Roselyne Tchoua, Jay Lofstead, Ron Oldfield, Manish Parashar, Nagiza Samatova, Karsten Schwan, Arie Shoshani, Matthew Wolf, Kesheng Wu, Weikuan Yu

Research output: Contribution to journal › Article › peer-review

184 Scopus citations

Abstract

Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

Original language	English
Pages (from-to)	1453-1473
Number of pages	21
Journal	Concurrency and Computation: Practice and Experience
Volume	26
Issue number	7
DOIs	https://doi.org/10.1002/cpe.3125
State	Published - May 2014

Funding

Funders	Funder number
National Science Foundation	0906324

Keywords

I/O middleware
high performance I/O
high performance computing

Access to Document

10.1002/cpe.3125

Cite this

Liu, Q., Logan, J., Tian, Y., Abbasi, H., Podhorszki, N., Choi, J. Y., Klasky, S., Tchoua, R., Lofstead, J., Oldfield, R., Parashar, M., Samatova, N., Schwan, K., Shoshani, A., Wolf, M., Wu, K., & Yu, W. (2014). Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks. Concurrency and Computation: Practice and Experience, 26(7), 1453-1473. https://doi.org/10.1002/cpe.3125

@article{3872fd165118443e88231ca7a7c8d152,

title = "Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks",

abstract = "Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.",

keywords = "I/O middleware, high performance I/O, high performance computing",

author = "Qing Liu and Jeremy Logan and Yuan Tian and Hasan Abbasi and Norbert Podhorszki and Choi, {Jong Youl} and Scott Klasky and Roselyne Tchoua and Jay Lofstead and Ron Oldfield and Manish Parashar and Nagiza Samatova and Karsten Schwan and Arie Shoshani and Matthew Wolf and Kesheng Wu and Weikuan Yu",

year = "2014",

month = may,

doi = "10.1002/cpe.3125",

language = "English",

volume = "26",

pages = "1453--1473",

journal = "Concurrency and Computation: Practice and Experience",

issn = "1532-0626",

publisher = "wiley",

number = "7",

}

Liu, Q, Logan, J, Tian, Y, Abbasi, H, Podhorszki, N , Choi, JY , Klasky, S, Tchoua, R, Lofstead, J, Oldfield, R, Parashar, M, Samatova, N, Schwan, K, Shoshani, A, Wolf, M, Wu, K & Yu, W 2014, 'Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks', Concurrency and Computation: Practice and Experience, vol. 26, no. 7, pp. 1453-1473. https://doi.org/10.1002/cpe.3125

TY - JOUR

T1 - Hello ADIOS

T2 - The challenges and lessons of developing leadership class I/O frameworks

AU - Liu, Qing

AU - Logan, Jeremy

AU - Tian, Yuan

AU - Abbasi, Hasan

AU - Podhorszki, Norbert

AU - Choi, Jong Youl

AU - Klasky, Scott

AU - Tchoua, Roselyne

AU - Lofstead, Jay

AU - Oldfield, Ron

AU - Parashar, Manish

AU - Samatova, Nagiza

AU - Schwan, Karsten

AU - Shoshani, Arie

AU - Wolf, Matthew

AU - Wu, Kesheng

AU - Yu, Weikuan

PY - 2014/5

Y1 - 2014/5

N2 - Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

AB - Applications running on leadership platforms are more and more bottlenecked by storage input/output (I/O). In an effort to combat the increasing disparity between I/O throughput and compute capability, we created Adaptable IO System (ADIOS) in 2005. Focusing on putting users first with a service oriented architecture, we combined cutting edge research into new I/O techniques with a design effort to create near optimal I/O methods. As a result, ADIOS provides the highest level of synchronous I/O performance for a number of mission critical applications at various Department of Energy Leadership Computing Facilities. Meanwhile ADIOS is leading the push for next generation techniques including staging and data processing pipelines. In this paper, we describe the startling observations we have made in the last half decade of I/O research and development, and elaborate the lessons we have learned along this journey. We also detail some of the challenges that remain as we look toward the coming Exascale era.

KW - I/O middleware

KW - high performance I/O

KW - high performance computing

UR - http://www.scopus.com/inward/record.url?scp=84899424893&partnerID=8YFLogxK

U2 - 10.1002/cpe.3125

DO - 10.1002/cpe.3125

M3 - Article

AN - SCOPUS:84899424893

SN - 1532-0626

VL - 26

SP - 1453

EP - 1473

JO - Concurrency and Computation: Practice and Experience

JF - Concurrency and Computation: Practice and Experience

IS - 7

ER -

Hello ADIOS: The challenges and lessons of developing leadership class I/O frameworks

Abstract

Funding

Keywords

Access to Document

Other files and links

Fingerprint

Cite this