Yes, the GPUs are on the same computing node, all GPUs are of the same type. What I mean by not synced is if I call tensor.data.grad after the backward call in the main worker they differ on each GPU. Which essentially means each GPUs are training separately but not really syncing the gradients at any point.
I’m afraid I cannot post the entire code and a minimal example might not capture the problem. That’s why I’m only asking for possible reasons in general.