@mrshenli , @Olivier-CR the torch.distributed.run docs isn’t a tutorial but a great place to start: torchrun (Elastic Launch) — PyTorch 1.10.0 documentation 2
For a more out-of-the-box experience, TorchX is where we are trying to make distributed job launching much easier: Distributed — PyTorch/TorchX main documentation 1