Thanks again!
By “your code” I meant the code here 2 but I didn’t realise that the code actually called the evaluate function just before save_on_master, which was calling dist.barrier, makes sense now. I should have said the torch vision code that you linked.
I will convert my torch.save to the save_on_master function and call dict.barrier() before it in my code as well then.
Thanks for the small example with resnet as well, all clear now. I’m still new to multi GPU training but this helped me a lot.