Solved by Martun_Karapetyan in post #4 
              
              
                Thanks for the help. 
I checked the model parameters, they were in perfect sync. 
I had another stupid bug. I used ReduceLROnPlateau when the validation accuracy plateaued, but each process looked at the validation accuracy of its subset of data. 1st process reduced the learning rate first, the othe…