Hi,
I have 2 EC2 machines with 4 GPUs each. That makes 8 GPUs in total.
I want to train a PyTorch DeepLab in data-parallel over those 8 cards. What should I do:
A. Launch a DDP training with 2 scripts/processes (1 per node), each doing torch.nn.DataParallel to data-parallel within node on the 4 cards
B. Launch a DDP training with 8 scripts/processes (1 per GPU), each executing pure DDP + PyTorch code and using only 1 GPU (leaving DDP doing the allreduces).
In both options: how to launch the processes with torchrun/launch.py/torch.distributed: once for the whole cluster, from some remote client? Once per node? Once per GPU?..