Using cross-layer adaptations for dynamic data management in large scale coupled scientific workflows

As system scales and application complexity grow, managing and processing simulation data has become a signif-icant challenge. While recent approaches based on data staging and in-situ/in-transit data processing are promising, dynamic data volumes and distributions, such as those occurring in AMR-based simulations, make the efficient use of these techniques challenging. In this paper we propose cross-layer adaptations that address these challenges and respond at runtime to dynamic data management require-ments. Specifically we explore (1) adaptations of the spatial resolution at which the data is processed, (2) dynamic place-ment and scheduling of data processing kernels, and (3) dy-namic allocation of in-transit resources. We also exploit co-ordinated approaches that dynamically combine these adap-tations at the different layers. We evaluate the performance of our adaptive cross-layer management approach on the In-trepid IBM-BlueGene/P and Titan Cray-XK7 systems using Chombo-based AMR applications, and demonstrate its effectiveness in improving overall time-to-solution and in-creasing resource efficiency.