Awesome! Thank you, @albanD!
In Python, I import os to set CUDA_VISIBLE_DEVICES, like this.

But when I set to use use GPU2 as previous, It doesn’t work.

The error say “invalid device ordinal”. Then I realize that when I set the CUDA_VISIBLE_DEVICES, there are only the devices set is visible. For example, if I set GPU2 and GPU3 visible,

That means ‘cuda:0’ now refer to GPU2, while ‘cuda:1’ refer to GPU3. Therefore ‘cuda:2’ will make error.
Thanks again to @albanD! You help me a lot!
I would have replied to you soon, but these few day I can’t access to the forum.