I’m training a pretty standard WGAN-GP on MNIST, and I’m trying out much larger batch sizes since that seems to be the standard wisdom now. When I parallelize across multiple GPUs I get enormous losses compared to using just one GPU. If I initialize my networks with nn.DataParallel(model, [0]) then I get pretty normal functionality:
ITER  0: D cost 1.200, G cost 2.729
ITER  200: D cost -3.931, G cost 2.298
But if I use nn.DataParallel(model, [0, 1, 2])  to run across more GPUs, I get absurd numbers:
ITER  200: D cost 112856899584.0, G cost 456269.437
I’ve used dataparallel successfully before with classifiers and the tutorial DCGAN, so I have no idea what the issue is here. I’m not parallelizing the loss or doing anything different than initializing the networks in this way. Are there some caveats here (like with FP16) that I’m not aware of with DataParallel?
I can post a full code example, but its a hundred lines or so and I’d rather not start off the post like that.