I think the reason for this is explained in https://pytorch.org/docs/stable/notes/randomness.html 6.
There are some PyTorch functions that use CUDA functions that can be a source of non-determinism. One class of such CUDA functions are atomic operations, in particular atomicAdd, where the order of parallel additions to the same value is undetermined and, for floating-point variables, a source of variance in the result. PyTorch functions that use atomicAdd in the forward include torch.Tensor.index_add_(), torch.Tensor.scatter_add_(), torch.bincount().
An example of such addition is 1 + 2**100 - 2**100 which can be 1 or 0 depending on the order of operations.
One thing I am unsure about is why we don’t see similar behavior with  torch.sum()