If CUDA_VISIBLE_DEVICES=3 yields an error, you would have to check your code, since you are indeed using different devices.
Check the code for all .cuda(), to() and device usages and use the default device instead of different ones.