Ho yes, very likely the gradients are different as well.
“It has been big pain to get the same results” Yes unfortunately the floating point standard makes it very hard as it does not force a given value of a result, but just forces values to be close enough to them.