The problem is that
eventhough I specified certain gpus that can be shown with os.environ[“CUDA_VISIBLE_DEVICES”],
the program keeps using only first gpu.
(But other program works fine and other specified gpus are allocated well.
because of that, I think it is not nvidia or system problem.
nvidia-smi shows all gpus well and there’s no problem.
I didn’t have problem with allocating gpus with below codes before (except when the system is not working)
it works fine when I use
torch.device("cuda:other_gpu_id" if torch.cuda.is_available() else "cpu")

)
os.environ["CUDA_DEVICE_ORDER"]="PCI_BUS_ID"
os.environ["CUDA_VISIBILE_DEVICES"] = str(args.gpu)

I wrote that before running main function.
and it works fine for other programs in same system (pytorch 1.8.1 with cuda11.1).
I printed that args.gpu variable, and could see that the value is not “0”.
and I tried below
CUDA_VISIBLE_DEVICES=4,5 python script.py

but still only first gpu is allocated.
when I asked the question in stackoverflow,
I could get the answer that those “os.envrion~” work in tensorflow and keras based code and not for pytorch code
As I didn’t have problem running pytorch code with those specifying-gpu codes before,
I want to know what makes it work and not work.