High end scientific codes with computational I/o pipelines: Improving their end-to-end performance

Fang Zheng, Jianting Cao, Jai Dayal, Greg Eisenhauer, Karsten Schwan, Matthew Wolf, Hasan Abbasi, Scott Klasky, Norbert Podhorszki

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

4 Scopus citations

Abstract

This paper uses computational I/O pipelines to integrate computations into the I/O path that perform data analytics on the data generated by scientific simulations. A novel attribute is the use of a pluggable execution environment in which analysis tools can be orchestrated into a multi-stage pipeline for processing simulation output data. Performance considerations are addressed through the use of a high performance data transport. The approach is evaluated with the end-to-end performance of a Magnetohydrodynamics application at large scale.

Original languageEnglish
Title of host publicationPDAC'11 - Proceedings of the 2011 International Workshop on Petascal Data Analytics
Subtitle of host publicationChallenges and Opportunities, Co-located with SC'11
Pages23-28
Number of pages6
DOIs
StatePublished - 2011
Event2011 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities, Held in Cooperation with the IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC'11 - Seattle, WA, United States
Duration: Nov 14 2011Nov 14 2011

Publication series

NamePDAC'11 - Proceedings of the 2011 International Workshop on Petascal Data Analytics: Challenges and Opportunities, Co-located with SC'11

Conference

Conference2011 2nd International Workshop on Petascal Data Analytics: Challenges and Opportunities, Held in Cooperation with the IEEE/ACM International Conference for High Performance Computing, Networking, Storage, and Analysis, SC'11
Country/TerritoryUnited States
CitySeattle, WA
Period11/14/1111/14/11

Keywords

  • End-to-end performance
  • I/O
  • In-situ analytics
  • Pipeline

Fingerprint

Dive into the research topics of 'High end scientific codes with computational I/o pipelines: Improving their end-to-end performance'. Together they form a unique fingerprint.

Cite this