NetSolve/D: A massively parallel grid execution system for scalable data intensive collaboration

The persistent mood of exhilaration in the research community over exponential increases in the capacity of computational resources has been tempered recently by the realization that a torrential influx of data from instruments, sensors and simulations is growing even faster than the resources needed to analyze it. The impact of this "data deluge," challenging enough by itself, is exacerbated by the fact that many data intensive projects today involve teams of collaborators spread out across geographically and organizationally distinct sites. A system that addresses these conditions must enable a community of collaborators, distributed throughout the wide area, to get responsive answers to dynamic queries and analyses applied to terascale or larger data sets. In this paper we describe the NetSolve/D system architecture which is designed to achieve this goal of data intensive on-line computing.