I have 4 gpu and 3 are being used, but the 3rd one is empty, so I want to use it, as shown below:

Screen Shot 2021-10-21 at 10.39.09 PM1168×548 158 KB

Then in my script, I have:
device = torch.device("cuda:3" if torch.cuda.is_available() else "cpu")
However, this gives me a weird error message:
RuntimeError: CUDA error: invalid device ordinal
Then I try gpu cuda:0, 1, or 2, it correctly used the GPU but got an out of memory error. This makes sense because other programs are using them. What I don’t understand is that, why cuda:3 produce that error?
Then I changed to:
evice = torch.device(“cuda” if torch.cuda.is_available() else “cpu”)
And at command line i use:
CUDA_VISIBLE_DEVICES=3 python my_script.py
This works fine, but this doesn’t allow me to use nohup:
nohup CUDA_VISIBLE_DEVICES=3 python my_script.py &
Anyway, why. can’t I specify cuda:3 in the script without using the command-line option?