I was trying to switch models between GPUs. I solved this by putting all models in one module inheriting nn.Module and applying DataParallel on that. In the forward method definitions, I called/accessed the individual models (generator and discriminator).