Solved by smth in post #4 
              
              
                I could not figure out when the updates are shared between the different processes. 

The updates aren’t shared between processes at all. All processes are seeing the same memory for the weights, so regardless of who updates the weights, all of them see updated weights. 
On (1), yes you need valida…