I notice that I get different results when I use DataParallel compared to a single GPU run. When I execute the same code on 1 GPU, I get the same loss if I repeat that procedure (assuming a fixed random seed). For some reason though, the loss is ~5% different if I use DataParallel. Is that a bug or some design-workaround that is a consequence of summing up the gradients?