See:

  
    
    
    How to use DataParallel in backward? 
  
  
    if you notice the examples, DataParallel is not applied to the entire network + loss. It is only applied to part of the network. 
before adding DataParallel: 
network = features (conv layers) -> classifier (linear layers) 
error = loss_function(network(input), target) 
error.backward() 
After adding DataParallel: 
network = DataParallel(features (conv layers)) -> classifier (linear layers) 
error = loss_function(network(input), target) 
error.backward()