For MPI backend at least, MPI non-overtaking guarantee should take care of messages being received in the correct order: http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node54.htm#Node54 12, http://mpi-forum.org/docs/mpi-2.2/mpi22-report/node61.htm#Node61 6.