Having tried to implement essentially all the communication functions with large-count support, I must say that this post makes it sound much easier than it actually is. Send-Recv are easy but many of the collectives are not. Or try support supporting counts >2^31 without resorting to non-portable functions like MPI_TYPE_CREATE_STRUCT. If they count is prime, it's impossible.

And the multiple message approach is not suitable for nonblocking operations and some collectives if you want to have a drop-in replacement. Using MPI_IN_PLACE in large-count reductions appears to be very difficult.

Please see https://github.com/jeffhammond/BigMPI for my code. Be warned that it is a work-in-progress and untested functions may not work.

Note also that BigMPI has motivated multiple tickets for the MPI Forum. Links to these tickets can be found at the bottom of the BigMPI home page.