Hey @spabho


Can you please define the terms: 1) node, 2) process in the context you are using them?


We usually use one node/machine/server to represent one physical computer which can be equipped with multiple GPUs.
One process is in the context of process/thread.

If I want to train my model on 4 GPUs, do you call it 4 processes? or 1 process?

In this case, using 4 processes with DDP should give you the best performance.

Does  init_method  correspond to the address of my PC or to the GPU I’m accessing on a cluster?

It corresponds to the address of your PC. It is giving some information for the 4 DDP processes to perform rendezvous.

In this 2 tutorial, what were you referring to as machine?

Machines should always refer to node/server/machine/computer.
To be clear, there are three concepts involved in DDP training:


Node/Machine/Server: a physical computer that can contain multiple GPUs. It can also talk to other node through network.

GPU: just one GPU

Process: Each process should run its own DDP instance. Usually Each DDP instance should exclusively operate on one GPU (if your model can fit in one GPU), and DDP instances will talk to each other through network.

This 6 example might serve better as a starting point for DDP.