Solved by ptrblck in post #2 
              
              
                The parameter gradient should be averaged as the data gradient (in x) depends on the input data. 
Since you are training with a different batch on each GPU you wouldn’t need to synchronize the data gradient unless I’m missing something in your use case.