A type system for high performance communication and computation

The manner in which data is represented, accessed and transmitted has an affect upon the efficiency of any computing system. In the domain of high performance computing, traditional frameworks like MPI have relied upon a relatively static type system with a high degree of a priori knowledge shared among the participants. However, modern scientific computing is increasingly distributed and dynamic, requiring the ability to dynamically create multi-platform workflows, to move processing to data, and to perform both in situ and streaming data analysis. Traditional approaches to data type description and communication in middleware, which typically either require a priori agreement on data types, or resort to highly inefficient representations like XML, are insufficient for the new domain of dynamic science. This paper describes a different approach, using FFS, a middleware library that implements efficient manipulation of application-level data. FFS provides for highly efficient binary data communication, XML-like examination of unknown data, and both third-party and in situ data processing via dynamic code generation. All of these capabilities are fully dynamic at run-time, without requiring a priori agreements or knowledge of the exact form of the data being communicated or analyzed.