Asynchrony and Failure Masking via Pseudo-Local Process Recovery in MPI Applications

For parallel solvers susceptible to hardware-related failures, localizing recovery to the processes directly affected by the failure allows preserving asynchronous progress and exhibits 'failure masking' due to limited propagation of recovery delays. This results in improved scalability compared to global recovery which is a disproportionate response. However, localizing recovery from hard failures is challenging because such failures are not transparent to the MPI runtime, requiring reconstruction of the communication layers and of a consistent application state. In this work we present the process- and data-recovery concepts that enable the performance and scalability of localized recovery despite the inherently non-local nature of some recovery steps. We present design enhancements to existing resilience middleware-the Fenix library and MPI User-Level Failure Mitigation-to robustly support larger-scale execution and 'pseudo-local' checkpointing and recovery from many process failures. Using an example stencil solver with emulated hard failures we present an experimental evaluation, with runs on up to 1000 ranks subject to 100 process failures, which confirms that that pseudo-local recovery has significantly improved weak scaling compared to the roughly exponential slowdown of global recovery. Our work shows how fault tolerance infrastructure originally designed for global checkpoint/restart can be repurposed to enable greater efficiency in a resilience-aware application.