Interesting, yea I never really had nn.DataParallel working for me, I often found that it just hangs (during training) it could possibly be my machine but not too sure. I’ll probably give it another go soon along with nn.DistributedDataParallel.
The legendary @ptrblck may know something though!