I would love to be able to cheaply ask from every process in e.g. a communicator, if anything bad happened. I actually don't need anything more fine grained than that, but asking should be "relatively" fast, and scale to very large number of processes.

Why?

My apps use non-blocking communication exclusively. In a nutshell I have an "mpi-request" scheduler on its own thread, that stores a request, an identifier, and a call back to execute when the request is finished (which can schedule further non-blocking operations).

At some synchronization points my other threads just wait for all request with a given identifier to finish before moving on.

If a process stops responding it might take a while for everything to collapse (which wastes computing time, the core/hours just keep ticking).

I would love to be able to ask at those synchronization points e.g. every 10 seconds, if everything is all right or something bad happen.

A more fine grained solution would be to store a time stamp in the scheduler and, if after a certain amount of time an mpi request hasn't finished, ask its process if everything is all right.

In my applications if an MPI process dies (or anything bad happens) I have data loss. The only thing I can do is fail as fast as possible to save computing time and display/log an error message. Wether the error message is "something bad happened with MPI" or "process XXX died" its not really that relevant.