@cbalioglu shall I run python -m torch.distributed.run once for the whole cluster? or once per node? or once per GPU? also asked here How to map processes to GPU in DDP and how to launch the DDP cluster? 1