Solved by tom in post #2 
              
              
                If you print the difference: 
print(output2-output1 - torch.as_tensor(bias)) 
you see that it is 1e-8ish, so this is an effect of numerical precision. 
With floating points, you cannot expect two “algebraically equivalent” (i.e. the maths say it should be the same but calculated differently) to give…