This issue is fixed in version 0.5.0

  
    
    
    Moving model to specific also allocates memory on GPU 0 
  
  
    Oh ok. I understand now. 
version 0.4.1: Maybe, pytorch uses cuda:0 always by default? 
pytorch master 0.5.0a0+ab6afc2: I have verified that this issue is fixed. 
Sorry for many edits of the answer