After removing all the non determinism caused by the GPU, and printing the gradient after each loss on machine 2 and 3 on both CPU and GPU, I can see that the loss on all is identical, and the gradients on machine 2 CPU and GPU along with machine 3 CPU are identical. For some reason machine 3 GPU doesn’t have identical gradients.