just to add to @ptrblck’s answer, which version of pytorch are you using? There was a bug in <= 0.4.1 versions that pytorch uses small amount of memory in GPU-0 when you do not use torch.cuda.set_device() or CUDA_VISIBLE_DEVICES=x. If you are using version >= 0.5.0, this bug is fixed I guess.

  
    
    
    Moving model to specific also allocates memory on GPU 0 
  
  
    Oh ok. I understand now. 
version 0.4.1: Maybe, pytorch uses cuda:0 always by default? 
pytorch master 0.5.0a0+ab6afc2: I have verified that this issue is fixed. 
Sorry for many edits of the answer