Nice post, this is an interesting topic.

Since you are an expert, do you know whether MPI libraries are able to automatically coalesce messages? Some time ago I came across a parameter of the OpenMPI's MCA framework called "btl_openib_use_message_coalescing" which should do that. However I have troubles figuring out how such thing is concretely implemented. To me there should be a kind of timer that waits "enough time" for several messages to be in the send buffer and then dispatches them all together to the receiver... but this would have a very bad impact on latency... and I don't think this would even be allowed in MPI.

So the question remains!