TY - GEN
T1 - Improving I/O performance through the dynamic remapping of object sets
AU - Logan, Jeremy
AU - Dickens, Phillip
PY - 2009
Y1 - 2009
N2 - Our research has been investigating a new approach to parallel I/O based on what we term objects. The premise of this research is that the primary obstacle to scalable I/O is the legacy view of a file as a linear sequence of bytes. The problem is that applications rarely access their data in a way that conforms to this data model, using instead what may be termed an object model, where each process accesses a (perhaps disjoint) collection of objects. We have developed an object-based caching system that provides an interface between MPI applications and a more powerful object file model, and have demonstrated significant performance gains based on this new approach. In this paper, we further explore the advantages that can be gained from using object-based I/O. In particular, we demonstrate that parallel I/O based on objects (termed parallel object I/O) can be dynamically remapped. That is, one application can output an object stream based on one object set, this can be captured and translated into a different object set that is more appropriate for another application. We demonstrate how such remapping can be accomplished, and provide an example application showing that using this technique can significantly improve I/O performance.
AB - Our research has been investigating a new approach to parallel I/O based on what we term objects. The premise of this research is that the primary obstacle to scalable I/O is the legacy view of a file as a linear sequence of bytes. The problem is that applications rarely access their data in a way that conforms to this data model, using instead what may be termed an object model, where each process accesses a (perhaps disjoint) collection of objects. We have developed an object-based caching system that provides an interface between MPI applications and a more powerful object file model, and have demonstrated significant performance gains based on this new approach. In this paper, we further explore the advantages that can be gained from using object-based I/O. In particular, we demonstrate that parallel I/O based on objects (termed parallel object I/O) can be dynamically remapped. That is, one application can output an object stream based on one object set, this can be captured and translated into a different object set that is more appropriate for another application. We demonstrate how such remapping can be accomplished, and provide an example application showing that using this technique can significantly improve I/O performance.
KW - Data-intensive applications
KW - High Performance Computing
KW - MPI-IO
KW - Parallel I/O
UR - http://www.scopus.com/inward/record.url?scp=74549187497&partnerID=8YFLogxK
U2 - 10.1109/IDAACS.2009.5342985
DO - 10.1109/IDAACS.2009.5342985
M3 - Conference contribution
AN - SCOPUS:74549187497
SN - 9781424448821
T3 - Proceedings of the 5th IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS'2009
SP - 259
EP - 265
BT - Proceedings of the 5th IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems
T2 - 5th IEEE International Workshop on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications, IDAACS'2009
Y2 - 21 September 2009 through 23 September 2009
ER -