If the relative error is in the range ~1e-6 the difference is most likely due to the limited precision used in float32 and you should get a lower error using e.g. flaot64.
The better way to compare floating point numbers is to use torch.allclose, which uses a small eps value to compare both tensors.