DataParallel imbalanced memory usage 
  
  
    Hi there, 
I’m going to re-edit the whole thread to introduce a unlikely behavior with DataParallel 
Right now there are several recent posts about this topic and I would like to summarize the problem. 

 

 

 
Right now it seems there is an imbalaced usage of GPUs when calling DataParallel. 
          
            [image]
          
 
From my experience and other users’ explanations I will explain why this happens: 
Using DataParallel there are 4 types of data to consider: 

Input
Output
Groun…