In reply to anon.

Agreed; a cheap cleanup is frequently the best way to go.

Many MPI implementations do this already, but without intervention from the application. E.g., if the MPI implementation detects a process failure before it calls MPI_FINALIZE, it decides that the entire job should be killed, and does so.

This works well when an individual MPI process has unexpectedly dies, but gets complicated if an entire node unexpectedly drops offline (e.g., someone kicks out a power cord). In that case, some kind of heartbeat mechanism is needed -- but care has to be taken that such a heartbeat mechanism scales and does not steal valuable network resources from the MPI application itself.