Distributed Training, Model Parallelism, Data Parallelism, Pytorch, Aurorain.
applications should spawn multiple processes and create a DDP instance per process. DDP uses collective communications in the torch.distributed package to synchronize gradients and buffers. Furthermore, DDP registers an autograd hook for each parameter from model.parameters(), and it will fire when