Are you only seeing this slowdown using nn.DataParallel or also a single GPU?
In the latter case, are you also seeing an increased memory usage?
If that’s the case, you might accidentally store some tensors which are still attached to the computation graph, or somehow extend the computation graph in each iteration.