It is a resnet model.
Each GPU has different mini batch mean and var for BatchNorm, but I fixed them by setting it to eval() mode just for the purpose of figuring out what’s going wrong.
Still the activations and gradients were similar but not the same between single and multi gpu on the order of 1e-2.