Kevinkevin189:

From my perspective, the spawn fcn start a job by initializing a group of procs with arg(rank,cfg), which means a groups of main_worker funtion. so every single main_worker,as a process, maintains a trainer object.But I see insde the trainer object, dist package also init process group to init a group of processes. What is the difference?where lies the true process?are there two hirearchy groups?


torch.mp.spawn spawns the actual processes, init_process_group doesn’t create any new processes but just initializes the distributed communication between spawned processes. For example if you spawn 4 processes using mp.spawn and call init_process_group on those 4 processes, init_process_group would ensure all 4 processes discover each other and now you can run collective calls like allreduce among those processes.



 Kevinkevin189:

the dist.init_process_method reports runtime errors.When I choose nccl as backend,it runs and waits until reporting (no nccl module),How to install an nccl module?


You can find the NCCL installation guide here: https://docs.nvidia.com/deeplearning/nccl/install-guide/index.html



 Kevinkevin189:

The init_process_group receives an argument init_method, which I can hardly understand,I leave it None but it bugged a message (No rendezvous handler for env://) ,I followed the tutorial to set environments variables MASTER_ADDR MASTER_PORT, it seems no effect.So I have to change it to ‘file://./sharefile’,what is this ?I totoally messed.


This section of our docs goes over details of init_method and how to use it: https://pytorch.org/docs/stable/distributed.html#tcp-initialization 3. In particular setting MASTER_ADDR and MASTER_PORT and leaving init_method=None should work.



 Kevinkevin189:

Now I’m running the code on my laptop,a single machine node with 1 GPU.But I need to train my model on a single machine node with multiple GPU cards.Is there any code to change?


You would need some changes where you need to one process per GPU on your single machine node. You can use mp.spawn to create one process per GPU.