DataParallel will chunk your data across the batch dimension so that it can send separate chunks to each GPU, which can explain the difference in gradients. Without seeing more of your model it is hard to say why this causes it not to train anymore.