In the meantime while you try to create a code snippet, we could continue the discussion and try to figure out, what’s going on.

If you use DataParallel with 2 GPUs, do you also double the batch size?
What batch size are you using? (Single and multiple GPU)
Do you have any BatchNorm layers?
Does your loss increase immediately after the first iteration or does it decrease for a period of time?