Hi @AndreaSottana
I don’t know which one you referred to as “your code”, but I suggested calling dist.barrrier() before torch.save in the code you showed above as follows:

For instance, I’d put torch.distributed.barrier() right before torch.save in your code above to wait for all the iterations in the epoch done before saving the model.

The example code actually calls dist.barrier() before save_on_master, and that happens in evaluate function 1, which is called before save_on_master

I’ve also noticed that when you save the model, you save the model_without_ddp.state_dict() instead of simply model.state_dict() . I’ve not seen this done before. Is there any advantage in doing this over simply saving the DDP model’s state dict?

Since DDP and DP will wrap your model, the state_dict saved by model.stage_dict() cannot be directly loaded a model without DDP/DP.
e.g.,
Try this minimal example
from torchvision import models

model = models.resnet18()
print(model.state_dict().keys())

model = torch.nn.parallel.DistributedDataParallel(model)
# or use DP instead of DDP
# model = torch.nn.DataParallel(model)
print(model.state_dict().keys())

From the first print statement, you’ll see pure module paths used in ResNet-18. But from the second print statement, you can confirm “module.” as a prefix for all the module paths shown by the first print statement.
It can be confirmed that your model is referred to as module in DP and DDP 1 implementations