XEMEM: Efficient shared memory for composed applications on multi-os/r exascale systems

Current trends in exascale systems research indicate that heterogeneity will abound in both the hardware and software layers on future HPC systems. It is our position that exascale environments are likely to be constructed from indepen-dent partitions of hardware and system software called enclaves, with multiple enclaves co-located on the same physical nodes and each executing an optimized operating sys-tem and runtime (OS/R) to support a particular application behavior. Fully utilizing these systems will require the ability to execute composed workloads, such as in situ applications, whereby HPC simulations execute synchronously with co-located analytic packages that in turn process simulation output via shared memory. In this work, we present the design and implementation of XEMEM, a shared memory system that can efficiently construct memory mappings across enclave OSes to support composed workloads while al-lowing diverse application components to execute in strictly isolated enclaves. By utilizing modifications to the Kitten lightweight kernel and Palacios lightweight virtual machine monitor, as well as leveraging our recent work on lightweight \co-kernels," we demonstrate that our approach can sup-port a diverse range of native and virtualized environments likely to be deployed on future exascale systems. Finally, we demonstrate that a multi-enclave system can reduce cross-workload contention and improve performance for a sample composed benchmark compared to a single OS approach.