Hey @Olivier-CR

A. Launch a DDP training with 2 scripts/processes (1 per node), each doing torch.nn.DataParallel to data-parallel within node on the 4 cards

DataParallel is single-machine multi-GPU. It won’t work in the multi-machine scenario. DistributedDataParallel is the appropriate feature to use.

B. Launch a DDP training with 8 scripts/processes (1 per GPU), each executing pure DDP + PyTorch code and using only 1 GPU (leaving DDP doing the allreduces).

Yep, this should work. One caveat is that you need to make sure each DDP process exclusively operate on a dedicated GPU. You can do this by either setting the CUDA_VISIBLE_DEVICES to different GPU for different processes, or use the local_rank within each process. See: Distributed communication package - torch.distributed — PyTorch master documentation 1

In both options: how to launch the processes with torchrun/launch.py/torch.distributed: once for the whole cluster, from some remote client? Once per node? Once per GPU?..

If you are using torch run.py/launch.py, you just need to do it once for each machine. If you directly call your user script without run/launch, you will need to do that once for each GPU.
I do recommend using TorchElastic 1 to launch jobs, as that will also provide failure recoveries.
Hey @Kiuk_Chung what’s the best tutorial to get started with run/launch? Thanks!