I have tried that as well but it didnt work for me with this class, it did however work when I was doing an nn.Sequential based model on the same computer. This is how I load my data, to be double sure I also do a prep function which essentially does
model = nn.DataParallel(model).cuda()
but is written like this
def prep(model):
’’’ Convert the model for GPU ‘’'
if torch.cuda.device_count() > 1:
model = torch.nn.DataParallel(model,
device_ids=range(torch.cuda.device_count())).cuda()
cudnn.benchmark = True
return model`
and my model is initialized as follows:
''' Prepare for Cuda '''
if args.cuda:
    discriminator = prep(skipgram_discriminator(vocabulary_size=args.vsize,\
            embedding_size=args.esize, pretrained=pretrained_embeddings))
    generator = prep(skipgram_generator(vocabulary_size=args.vsize,\
            embedding_size=args.esize))

And the data for a forward pass is initialized like this looks like this:
mb_iwords_gpu = Variable((mb_iword.long().cuda() if args.cuda else mb_iword))
mb_cwords_gpu = Variable((mb_cword.long().cuda() if args.cuda else mb_cword))
But yet this still doesnt allocate any memory on the second GPU. I hope this clarifies the issue a bit more. Any help is greatly appreciated.