Thanks for the reply. That could be the case but in all the examples I’ve seen using distributed launch didn’t show how to properly save the checkpoint. When I save the checkpoint I was just using torch.save. Should I be using something else?



 ptrblck:

It seems you’ve saved some checkpoints in the previous runs without making sure only a single process (e.g. rank0) writes to the files.


How do I do that? Any pointers or examples where to look would be much appreciated!