Stream processing for near real-time scientific data analysis

The demand for near real-time analysis of streaming data is increasing rapidly in scientific projects. This trend is driven by the fact that it is expensive and time consuming to design and execute complex experiments and simulations. During an experiment, the research team and the team at the experiment facility will want to analyze data as it is generated, interpret it, and collaboratively make decisions to modify the experiment parameters or abort the experiment in order to prevent events that may damage experimental instruments or to avoid wasting resources if there is a problem. The increasing velocity and volume of streaming data and the multi-institutional nature of large-scale scientific projects present challenges to near real-time analysis of streaming data. In this work we develop a framework to address these challenges. This framework provides an interface for applications to define and interact with named, self-describing streams, takes advantage of advanced network technologies, and implements support for the reduction and compression of data at the source. We describe this framework and demostrate its application in three scientific applications.