Make sure to set the visible devices before any other imports.
The easiest way would be to run your whole script using:
CUDA_VISIBLE_DEVICES=3 python script.py args

Also note, that setting the visible devices to a particular device masks all other devices.
That means you should only see the one device now with id 0 in your python script.
torch.cuda.set_device(3) shouldn’t work therefore.