Question: I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Answer: 
One use of `MPI_Barrier` is for example to control access to an external resource such as the filesystem, which is not accessed using MPI. For example, if you want each process to write stuff to a file in sequence, you could do it like this:

```
int rank, size;
MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);
for ( int ii = 0; ii < size; ++ii ) {
    if ( rank == ii ) {
        // my turn to write to the file
        writeStuffToTheFile();
    }
    MPI_Barrier(MPI_COMM_WORLD);
}

```

That way, you can be sure that no two processes are concurrently calling `writeStuffToTheFile`.
May MPI\_Barrier() is not often used, but it is useful.
In fact, even if you were use the synchronous communication, the MPI\_Send/Recv() can only make sure the two processes is synchronized.
In my project, a cuda+MPI project, all i used is asynchronous communication.
I found that in some cases if i dont use the MPI\_Barrier() followed by the Wait() function, the situation that two processes(gpu) want to transmit data to each other at the same time is very likely to happen, which could badly reduce the program efficiency.
The bug above ever divers me mad and take me a few days to find it. Therefore you may think carefully whether use the MPI\_Barrier() when you used the MPI\_Isend/Irecv in your program. Sometimes sync the processes is not only neccessary but also MUST, especially ur program is dealing with the device.