Nested workflows for loosely coupled HPC simulations

The increasing complexity of modern scientific simulations has given rise to the notion of re-usability as a means to reduce development time and effort. One form of re-usability involves hierarchical modelling, where the code and artifacts that model a physical phenomenon are used as the building blocks for more complex coupled physical systems. This reusability mode allows for improvements in constituents sub-models to impact the overall fidelity of the entire system with minimal effort. This benefit, however, hinges upon the ability to provide a fairly loose coupling across model boundaries, where changes in any sub-model do not result in wholesale changes to other sub-models, or in the structure and details of the code for the entire physical system. In this paper, we present the design, implementation, and case studies for incorporating sub-workflows as building blocks in a framework for loosely coupled high performance simulations. We outline the issues involved in providing lightweight customization points for full workflows that allows their use unchanged in a nested-workflow setting, while providing each sub-workflow with a separate execution context that ensures non-interference with other parts of the simulation. We present several use cases that demonstrate the successful use of the proposed design and its flexibility in accommodating different customization and scaling requirements.