I could not figure out when the updates are shared between the different processes.

The updates aren’t shared between processes at all. All processes are seeing the same memory for the weights, so regardless of who updates the weights, all of them see updated weights.
On (1), yes you need validation once in a while, so not ideal. (2) should actually be fine to do, practically – train time is much greater than process sync time. (4) is fine, but all the other processes have to stop training while one of the process is doing the testing, or else you have the situation where – “while testing in process 0, process 1 updates weights, hence testing network for validation sample 0 is not the same as testing network for validation sample 1”