TY - GEN
T1 - Lightweight I/O for scientific applications
AU - Oldfield, Ron A.
AU - Ward, Lee
AU - Riesen, Rolf
AU - Maccabe, Arthur B.
AU - Widener, Patrick
AU - Kordenbrock, Todd
PY - 2006
Y1 - 2006
N2 - Today's high-end massively parallel processing (MPP) machines have thousands to tens of thousands of processors, with next-generation systems planned to have in excess of one hundred thousand processors. For systems of such scale, efficient I/O is a significant challenge that cannot be solved using traditional approaches. In particular, general purpose parallel file systems that limit applications to standard interfaces and access policies do not scale and will likely be a performance bottleneck for many scientific applications. In this paper, we investigate the use of a "lightweight" approach to I/O that requires the application or I/O-library developer to extend a core set of critical I/O functionality with the minimum set of features and services required by its target applications. We argue that this approach allows the development of I/O libraries that are both scalable and secure. We support our claims with preliminary results for a lightweight checkpoint operation on a development cluster at Sandia.
AB - Today's high-end massively parallel processing (MPP) machines have thousands to tens of thousands of processors, with next-generation systems planned to have in excess of one hundred thousand processors. For systems of such scale, efficient I/O is a significant challenge that cannot be solved using traditional approaches. In particular, general purpose parallel file systems that limit applications to standard interfaces and access policies do not scale and will likely be a performance bottleneck for many scientific applications. In this paper, we investigate the use of a "lightweight" approach to I/O that requires the application or I/O-library developer to extend a core set of critical I/O functionality with the minimum set of features and services required by its target applications. We argue that this approach allows the development of I/O libraries that are both scalable and secure. We support our claims with preliminary results for a lightweight checkpoint operation on a development cluster at Sandia.
UR - http://www.scopus.com/inward/record.url?scp=46049092568&partnerID=8YFLogxK
U2 - 10.1109/CLUSTR.2006.311853
DO - 10.1109/CLUSTR.2006.311853
M3 - Conference contribution
AN - SCOPUS:46049092568
SN - 1424403286
SN - 9781424403288
T3 - Proceedings - IEEE International Conference on Cluster Computing, ICCC
BT - 2006 IEEE International Conference on Cluster Computing, Cluster 2006
T2 - 2006 IEEE International Conference on Cluster Computing, Cluster 2006
Y2 - 25 September 2006 through 28 September 2006
ER -