Bone:

0.009999999776482582 rank0
-0.0011623682221397758 rank1
-0.0011623682221397758 rank2
-0.0011623682221397758 rank3


Unless avg_acc are already synced on rank1-3, otherwise it looks like rank0 has successfully broadcast to the other three ranks (so that they share the same value) and then the avg_acc was somehow modified afterwards before prints.
Can you try add a stream sync after broadcast and print immediately after that? Sth like:
dist.broadcast(avg_acc, src=0)
torch.cuda.current_stream(device).synchronize()
print(avg_acc)

BTW, would I be correct that each process/rank exclusively occupies one device?