In reply to Jeff Squyres.

The "challenge" is having 2 MPI process do the following in a completely non-blocking way (I just use non-blocking send and recv for exposition, but a similar example using MPI_IProbe could be constructed):

Process A:
non-blocking sends size of array
non-blocking send array

Process B:
non-blocking receive size of array
non-blocking allocate buffer
non-blocking receive array on buffer

In C++ pseudo-code:

Process A:
std::vector data {1, 2, 3, 4};
auto request
= nb_send(process_b, tag_0, data.size())
.then([]() { // schedules a continuation:
nb_send(process_b, tag_1, data.data(), data.size());
});
// application code folows {
// not a single call to mpi test in here:
// the second send is scheduled as soon as the first
// completes
// }
request.get(); // blocks until both sends complete

Process B:
std::size_t size;
std::vector buffer;

auto request
= nb_recv(process_a, tag_0, &size)
.then([&]() { // schedules continuation
// allocate memory for the buffer before the second recv
buffer.resize(size);
nb_recv(process_a, tag_1, buffer.data(), size);
});
// application code {
// not a single call to test in here:
// as soon as the first recv completes, the continuation
// runs asynchronously and memory is allocated
// and the second recv is scheduled
// }
request.get(); // blocks until both recvs complete

The main net win is that one fires all the communication logic at the beginning, and blocks at the very end. That is, the unrelated application code in the middle doesn't need to e.g. call a function "every now and then" to just advance the continuation.

I don't see how to do this without spawning a different thread that continuously loops over the requests calling MPI test to advance to the next continuation.

If one has to spawn a different thread anyways the advantage of non-blocking MPI calls is still there, but it is less important (they allow cool things like having a pool of request + continuations).

P.S: One could just have sent the array at once with send, but the receive end would still need to repeatedly call MPI_IProbe before executing the continuation (I just chose the two sends + two receives for exposition purposes).