TY - GEN
T1 - A system-aware optimized data organization for efficient scientific analytics
AU - Tian, Yuan
AU - Klasky, Scott
AU - Yu, Weikuan
AU - Abbasi, Hasan
AU - Wang, Bin
AU - Podhorszki, Norbert
PY - 2012
Y1 - 2012
N2 - Large-scale scientific applications on High End Computing systems produce a large volume of highly complex datasets. Such data imposes a grand challenge to conventional storage systems for the need of efficient I/O solutions during both the simulation runtime and data post-processing phases. With the mounting needs of scientific discovery, the read performance of large-scale simulations has becomes a critical issue for the HPC community. In this study, we propose a system-aware optimized data organization strategy that can organize data blocks of multidimensional scientific data efficiently based on simulation output and the underlying storage systems, thereby enabling efficient scientific analytics. Our experimental results demonstrate a performance speedup up to 72 times for the combustion simulation S3D, compared to the logically contiguous data layout.
AB - Large-scale scientific applications on High End Computing systems produce a large volume of highly complex datasets. Such data imposes a grand challenge to conventional storage systems for the need of efficient I/O solutions during both the simulation runtime and data post-processing phases. With the mounting needs of scientific discovery, the read performance of large-scale simulations has becomes a critical issue for the HPC community. In this study, we propose a system-aware optimized data organization strategy that can organize data blocks of multidimensional scientific data efficiently based on simulation output and the underlying storage systems, thereby enabling efficient scientific analytics. Our experimental results demonstrate a performance speedup up to 72 times for the combustion simulation S3D, compared to the logically contiguous data layout.
KW - Data Layout
KW - I/O
UR - http://www.scopus.com/inward/record.url?scp=84863917800&partnerID=8YFLogxK
U2 - 10.1145/2287076.2287095
DO - 10.1145/2287076.2287095
M3 - Conference contribution
AN - SCOPUS:84863917800
SN - 9781450308052
T3 - HPDC '12 - Proceedings of the 21st ACM Symposium on High-Performance Parallel and Distributed Computing
SP - 125
EP - 126
BT - HPDC '12 - Proceedings of the 21st ACM Symposium on High-Performance Parallel and Distributed Computing
T2 - 21st ACM Symposium on High-Performance Parallel and Distributed Computing, HPDC '12
Y2 - 18 June 2012 through 22 June 2012
ER -