Abstract
Resiliency is and will be a critical factor in determining scientific productivity on current and exascale supercomputers, and beyond. Applications oblivious to and incapable of handling transient soft and hard errors could waste supercomputing resources or, worse, yield misleading scientific insights. We introduce a novel application-driven silent error detection and recovery strategy based on application health monitoring. Our methodology uses application output that follows known patterns, as indicators of an application's health and knowledge that violation of these patterns could be indication of faults. Information from system monitors that report hardware and software health status is used to corroborate faults. Collectively, this information is used by a fault coordinator agent to take preventive and corrective measures by applying computational steering to an application between checkpoints. This cooperative fault management system uses the Fault Tolerance Backplane as a communication channel. The benefits of this framework are demonstrated with two real application case studies, molecular dynamics, and quantum chemistry simulations, on scalable clusters with simulated memory and I/O corruptions. The developed approach is general and can be easily applied to other applications.
Original language | English |
---|---|
Article number | e5449 |
Journal | Concurrency and Computation: Practice and Experience |
Volume | 32 |
Issue number | 2 |
DOIs | |
State | Published - Jan 25 2020 |
Funding
Financial support for this work was provided by National Institute of General Medical Sciences of the National Institutes of Health (under award numbers R21GM083946 and R01GM105978) to PKA, and the US Department of Energy, Office of Science, Office of Advanced Scientific Computing Research through the project on Coordinated and Improved Fault Tolerance for High Performance Computing Systems (CIFTS) project. We thank various members of the CIFTS team, Sadaf Alam, Christian Engelmann, and Hai Ah Nam, for discussions and feedback.
Keywords
- exascale resiliency
- fault tolerance
- heterogeneous systems
- molecular dynamics
- quantum chemistry calculations
- silent errors