Question: I wonder when do I need to use barrier? Do I need it before/after a scatter/gather for example? Or should OMPI ensure all processes have reached that point before scatter/gather-ing? Similarly, after a broadcast can I expect all processes to already receive the message?

Answer: 
All collective operations in MPI before MPI-3.0 are blocking, which means that it is safe to use all buffers passed to them after they return. In particular, this means that all data was received when one of these functions returns. (However, it does not imply that all data was sent!)
So MPI\_Barrier is not necessary (or very helpful) before/after collective operations, if all buffers are valid already.

Please also note, that MPI\_Barrier does not magically wait for non-blocking calls. If you use a non-blocking send/recv and both processes wait at an MPI\_Barrier after the send/recv pair, it is not guaranteed that the processes sent/received all data after the MPI\_Barrier. Use MPI\_Wait (and friends) instead. So the following piece of code contains errors:

```
/* ERRORNOUS CODE */

Code for Process 0:
Process 0 sends something using MPI_Isend
MPI_Barrier(MPI_COMM_WORLD);
Process 0 uses buffer passed to MPI_Isend // (!)

Code for Process 1:
Process 1 recvs something using MPI_Irecv
MPI_Barrier(MPI_COMM_WORLD);
Process 1 uses buffer passed to MPI_Irecv // (!)

```

Both lines that are marked with `(!)` are unsafe!

MPI\_Barrier is only useful in a handful of cases. Most of the time you do not care whether your processes sync up. Better read about blocking and non-blocking calls!
May MPI\_Barrier() is not often used, but it is useful.
In fact, even if you were use the synchronous communication, the MPI\_Send/Recv() can only make sure the two processes is synchronized.
In my project, a cuda+MPI project, all i used is asynchronous communication.
I found that in some cases if i dont use the MPI\_Barrier() followed by the Wait() function, the situation that two processes(gpu) want to transmit data to each other at the same time is very likely to happen, which could badly reduce the program efficiency.
The bug above ever divers me mad and take me a few days to find it. Therefore you may think carefully whether use the MPI\_Barrier() when you used the MPI\_Isend/Irecv in your program. Sometimes sync the processes is not only neccessary but also MUST, especially ur program is dealing with the device.