TY - GEN
T1 - Exploring Automatic, Online Failure Recovery for Scientific Applications at Extreme Scales
AU - Gamell, Marc
AU - Katz, Daniel S.
AU - Kolla, Hemanth
AU - Chen, Jacqueline
AU - Klasky, Scott
AU - Parashar, Manish
N1 - Publisher Copyright:
© 2014 IEEE.
PY - 2014/1/16
Y1 - 2014/1/16
N2 - Application resilience is a key challenge that must be addressed in order to realize the exascale vision. Process/node failures, an important class of failures, are typically handled today by terminating the job and restarting it from the last stored checkpoint. This approach is not expected to scale to exascale. In this paper we present Fenix, a framework for enabling recovery from process/node/blade/cabinet failures for MPI-based parallel applications in an online (i.e., Without disrupting the job) and transparent manner. Fenix provides mechanisms for transparently capturing failures, re-spawning new processes, fixing failed communicators, restoring application state, and returning execution control back to the application. To enable automatic data recovery, Fenix relies on application-driven, diskless, implicitly coordinated check pointing. Using the S3D combustion simulation running on the Titan Cray-XK7 production system at ORNL, we experimentally demonstrate Felix's ability to tolerate high failure rates (e.g., More than one per minute) with low overhead while sustaining performance.
AB - Application resilience is a key challenge that must be addressed in order to realize the exascale vision. Process/node failures, an important class of failures, are typically handled today by terminating the job and restarting it from the last stored checkpoint. This approach is not expected to scale to exascale. In this paper we present Fenix, a framework for enabling recovery from process/node/blade/cabinet failures for MPI-based parallel applications in an online (i.e., Without disrupting the job) and transparent manner. Fenix provides mechanisms for transparently capturing failures, re-spawning new processes, fixing failed communicators, restoring application state, and returning execution control back to the application. To enable automatic data recovery, Fenix relies on application-driven, diskless, implicitly coordinated check pointing. Using the S3D combustion simulation running on the Titan Cray-XK7 production system at ORNL, we experimentally demonstrate Felix's ability to tolerate high failure rates (e.g., More than one per minute) with low overhead while sustaining performance.
UR - https://www.scopus.com/pages/publications/84936933249
U2 - 10.1109/SC.2014.78
DO - 10.1109/SC.2014.78
M3 - Conference contribution
AN - SCOPUS:84936933249
T3 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC
SP - 895
EP - 906
BT - Proceedings of SC 2014 - International Conference for High Performance Computing, Networking, Storage and Analysis
PB - IEEE Computer Society
T2 - International Conference for High Performance Computing, Networking, Storage and Analysis, SC 2014
Y2 - 16 November 2014 through 21 November 2014
ER -