Solved by mrshenli in post #2 
              
              
                Hey @TaranRai I think you need do call  model.to(device) before calling DataParallel ctor. 

Would this get the most out of the GPUS? 

DistributedDataParallel is expected to be faster than DataParallel. See this example