Ardeal:

how to specify rank number for each process when I use spawn function to start main_worker?


The method you start with mp.spawn must take in as its first argument a rank parameter (proc) in your example, which will be the rank of the process. Ranks are assigned in order of the processes starting in each worker.



 Ardeal:

how to specify/check local_rank of each process in main_worker?


After initializing the process group (as you have with init_process_group), you can use the dist.get_rank() API to get the global rank of the process in the world. To get the local rank, assuming a homogenous set up, mod the result of dist.get_rank() by the number of GPUs on the machine.



 Ardeal:

Question 3:
world_size means total number of GPUs used in the processes.
rank is the index of each processes/process number.
local_rank means the GPU index used in the process(rank).
is my understanding correct?


Your understanding of rank and local_rank is correct, though note that local_rank is not necessarily the GPU index. You can assign GPUs to ranks non-sequentially using torch.cuda.set_device() API, for example rank 0 could operate on GPU 1 and rank 1 could operate on GPU 0. Regarding world-size, world_size = total no. of active processes across all nodes. Usually this will be equal to the no. of GPUs available