Solved by royboy in post #2 
              
              
                Pytorch will create context on gpu 0 regardless of which gpus you end up using. You should set CUDA_VISIBLE_DEVICES. 
For more detailed discussion, this link is helpful https://github.com/pytorch/pytorch/issues/3477.