UPDATE:
so I’ve just tested the forward and backward passes and it looks like the activations and gradients are roughly the same, but they are on the order of 1e-2 to 1e-3 different between the single gpu and multi-gpu case… Could this be the problem?