Hi Py!



 pytorcher:

Is there any mode that will still allow decimal points but give exact numbers and the rounding differences just get cut off.


No.
There is neither a built-in way to do this, nor any practical way to
do it “by hand.”
But you  speak as if you are convinced that your “actual functions
are different,” even though the computational evidence you cited
in your earlier thread shows that your “actual functions” agree up
to floating-point round-off, which is as much as you can expect.
Floating-point round-off is real, and you can’t get rid of it.  But, in
general, it’s not a problem.  After one iteration, your two versions
agree up to floating-point round-off.  After a couple more iterations,
the floating-point round-off accumulates some.  And after a few
more iterations, it accumulates to the point that your optimizer’s
trajectory in parameter space starts to wander off along two
different paths.  But if you had run the same version twice, but
where you initialized your model to slightly different random
weights, your optimizer’s trajectory would also wander off along
two different paths.
The loss surface in parameter space has a bunch of very
complicated, high-dimensional peaks and valleys, and very
many – for practical purposes equivalent – approximate minima.
So it’s no surprise that slightly different initial conditions – or slightly
different round-off error-- will set you off along different paths to
different – but, in practice, equally good – approximate minima.
Without evidence that something is actually wrong, I think you’re
seeing problems where there are none.
Best.
K. Frank