Are you using any cuda() or to() calls inside your model (__init__ or forward)?
If so, could you remove them as they might create the device mismatch.
nn.DataParallel will automatically create model replicas for you and you don’t need to push internal model parameters to a specific device manually (only the nn.DataParallel model to the default device).