You might see rel. small errors due to the limited floating point precision, if the used algorithm is non-deterministic.
Could you set torch.set_deterministic(True) and rerun the script? Also, which PyTorch version are you using and CPU/GPU for this test?