That’s strange, the tensors should be identical after all_reduce, could you please post some more details about your setup (e.g., how you are running torch.distributed such as the backend being used)?