Hi @AndreaSottana

Yes, technically you can choose the rank you like as the main process. Since the rank number starts from zero, I’d suggest using rank 0 as the main process
Using DDP, the two processes synchronize gradient across processes. You can confirm the detail here 1. If you don’t control the timing of saving your model when allowing both the processes to save a model, it may cause file handler error like one of the processes attempts to overwrite a file whose save operation is not complete.
It can be controlled by torch.distributed.barrier(), which waits for other processes. For instance, I’d put torch.distributed.barrier() right before torch.save in your code above to wait for all the iterations in the epoch done before saving the model. The same example in torchvision 1 uses it in evaluation to compute global accuracy across the processes. The synchronize_between_processes() uses torch.distributed.barrier() as dist.barrier() 1