An Application Driven I/O Optimization Approach for PetaScale Systems and Scientific Discoveries

Application workflows typically involve large-scale simulations, applications, and subsequent analysis, verification and validation. One of the most important requirements shared by various applications running on petascale systems is fast, portable, scalable I/O which is componentized, metadata rich and easy to use. The Adaptable I/O System (ADIOS) serves as a high-level I/O interface for an application to select which I/O libraries and formats to use without changing the application program. Codes such as GTC generate hundreds of TBs of data on hundreds of thousands of cores in a twenty four hour period. One must therefore optimize the I/O both for fast output in the generation phase and for fast input in the analysis phase. Both the writing and reading efficiency of I/O are critical for knowledge discovery. Development of a high level software infrastructure to allow optimization of I/O for entire workflows (including High-Performance I/O when reading data with different patterns) would greatly improve end-to-end performance in the knowledge discovery cycle.

This project plans to develop efficient I/O methods which will enable application scientists to optimize data for writing, and which will be able to re-organize the data to obtain optimal performance for common reading patterns used by scientists. This project directly impacts the I/O performance of many petascale applications, including the GTC, GTS, XGC-1, Chimera, and S3D codes, and work directly with these teams to optimize the I/O in all stages of their scientific workflow.