I suspect that calling model.cuda() on nn.DataParallel is causing trouble. Can you try creating the model first, then calling model.cuda(), and then wrapping it in nn.DataParallel? I think that in this example the entire model is copied to GPU every time you call forward.