Hello
You’d need to initialize process group for DDP and it will synchronize workers for the first iteration.
Not sure why you are trying to call dist.all_reduce() here, normally DDP will handle model parameters for you.
Your scheduler is similar to optimizer in this context?

I think this is correct, I am unsure if the barrier is needed. I assume so because otherwise some processes may be lagging behind and not have updated val_acc yet.

You generally don’t need to synchronize your workers, e.g.
https://pytorch.org/tutorials/intermediate/ddp_tutorial.html
(barriers are only used to synchronize while checkpointing)