If you are using Pytorch 0.4 you could specify the device by doing
device = torch.device('cuda:0')
X = X.to(device)

Cuda:0 is always the first visible GPU. So if you set CUDA_VISIBLE_DEVICES (which I would recommend since pytorch will create cuda contexts on all other GPUs otherwise) to another index (e.g. 1), this GPU is referred to as cuda:0. Alternatively you could specify the device as torch.device('cpu') for running your model/tensor on CPU.
The global GPU index (which is necessary to set CUDA_VISIBLE_DEVICES in the right way) can be seen by the nvidia-smi command in the shell/bash.