If you print the difference:
print(output2-output1 - torch.as_tensor(bias))
you see that it is 1e-8ish, so this is an effect of numerical precision.
With floating points, you cannot expect two “algebraically equivalent” (i.e. the maths say it should be the same but calculated differently) to give exactly the same result, but the errors in the computation make it slightly different.
This gets even more tricky when parallel computation is involved which will not even necessarily compute the same result when run twice because the order might be “random” (look for reproducible/deterministic in the PyTorch documentation, the CUDA atomicAdd command with floating point is perhaps the most common source of this in PyTorch).
The most simple example perhaps is running 1e32+1-1e32 this is not the same as 1e32-1e32+1.
Best regards
Thomas