High end scientific codes with computational I/o pipelines: Improving their end-to-end performance

Fang Zheng, Jianting Cao, Jai Dayal, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Hasan Abbasi, Scott Klasky, Norbert Podhorszki

    Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

    4 Scopus citations

    Abstract

    This paper uses computational I/O pipelines to integrate computations into the I/O path that perform data analytics on the data generated by scientific simulations. A novel attribute is the use of a pluggable execution environment in which analysis tools can be orchestrated into a multi-stage pipeline for processing simulation output data. Performance considerations are addressed through the use of a high performance data transport. The approach is evaluated with the end-to-end performance of a Magnetohydrodynamics application at large scale.

    Original languageEnglish
    Title of host publicationPDAC'11 - Proceedings of the 2011 International Workshop on Petascal Data Analytics
    Subtitle of host publicationChallenges and Opportunities, Co-located with SC'11
    Pages23-28
    Number of pages6
    DOIs
    StatePublished - 2011
    Event2011 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities, Held in Cooperation with the IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC'11 - Seattle, WA, United States
    Duration: Nov 14 2011Nov 14 2011

    Publication series

    NamePDAC'11 - Proceedings of the 2011 International Workshop on Petascal Data Analytics: Challenges and Opportunities, Co-located with SC'11

    Conference

    Conference2011 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities, Held in Cooperation with the IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC'11
    Country/TerritoryUnited States
    CitySeattle, WA
    Period11/14/1111/14/11

    Keywords

    • End-to-end performance
    • I/O
    • In-situ analytics
    • Pipeline

    Fingerprint

    Dive into the research topics of 'High end scientific codes with computational I/o pipelines: Improving their end-to-end performance'. Together they form a unique fingerprint.

    Cite this