887574002:

Hi, thank you for your answer. Regarding your code, when we run the code isn’t it needed to give the gpu index? How dose coed understand on which gpus should run the process.


A process can access any visible GPU. The one-process-per-GPU requirement is from DDP to avoid NCCL comm hang. Ideally, we should set CUDA_VISIBLE_DEVICES for each process accordingly, so that each process only sees one GPU and cuda:0 on each process points to a different GPU. But if you are confident that no code would accidentally access a different GPU, directly doing .to(gpu) would be sufficient. We are using the id of the process provided by mp.spawn as the gpu id.

And my second question is a bout  num_replicas , shouldn’t it be equal to num_process?

Yep, you are right.