You are right @ngimel , my problem was that from conda I got the latest release 0.3.0 but the docs I was reading were already about version 0.4.0 
This commit should fix my problem:

  
      github.com/pytorch/pytorch
  
  
      
    
  


  Fix distributed dataloader so it pins memory to current GPU not GPU 0.




  by csarofeen
  on 06:58PM - 15 Dec 17



  changed 1 files
  with 7 additions
  and 3 deletions.