Open MPI coalesceses messages only when it is stalled from sending.

For example, when the hardware send queue is full, but the application is still sending more new (short) messages. If successive new messages in this case are of the same MPI signature (e.g., to the same receiver on the same CID with the same tag), then Open MPI will coalesce the messages together.

Hence, when the send queue eventually opens up, there will be fewer entries placed on the queue than MPI messages that are actually sent.

Make sense?

This was a particularly ugly feature to implement and debug. But I'd be surprised if other OpenFabrics-based MPI implementations don't do similar things (i.e., coalesce only when otherwise blocked from sending).