@MRGAO1996
Could you please format the code with proper indention?
If this line is with in the if local_rank == 0 block, then this is indeed wrong. You need to call all_reduce on all processes, as it is a collective communication.
if local_rank == 0:
    dist.all_reduce(lr_broadcast)