Solved by rvarm1 in post #2 
              
              
                Hi, 
I’m guessing this is because your update is conditioned on if local_rank == 0. If you only want to run this validation on rank 0, you can broadcast the result to the rest of the ranks as follows: 
if local_rank == 0:
    new_lr = compute_new_lr()
    dist.broadcast(torch.tensor(new_lr), src=0)
…