Thanks for your reply.
Yes, you are right, I can lunch the script with CUDA_VISIBLE_DEVICES=0,1.
What I’m trying to do is to set the GPU id manually within the script so that I can distribute the model to more than one GPU with nn.parallel.DistributedDataParallel(model, device_ids=[**more than one**])
For example, on node two I want to set the GPU for [0,1] for processes with global rank [2,3] and allow model parallel on GPU [2,3]. But if I use CUDA_VISIBLE_DEVICES=0,1, both process can only see two GPU instead of four, I can only set device_ids=[0]. When I tried to use torch.cuda.set_device(local_rank), it has the problem like I described above.
Let me know if you need to know anything.