Pytorch will create context on gpu 0 regardless of which gpus you end up using. You should set CUDA_VISIBLE_DEVICES.
For more detailed discussion, this link is helpful https://github.com/pytorch/pytorch/issues/3477 37.