Hi Py!



 pytorcher:

I don’t know when the differences are because of floating point rounding and when they are because there is actual functionality differences.


First, don’t look at differences in your results after training for a
number of (batch) iterations.  It is perfectly possible for your
model parameters to wander off in distinctly different, although
content-wise, broadly-equivalent directions, after accumulated
round-off errors start them down different paths.
You should look at whether your differences are compatible with
round-off error after passing a single batch through your model,
checking your output, your loss, and your gradients after
backpropagating once.  If you still think these differences exceed
reasonable, somewhat accumulated round-off error, trace
intermediate steps through the layers of your model.  Unless you
have an ill-conditioned Linear in there somewhere, no single
layer should give results that differ by more than a few times
round-off error.
The next general check you can make is to move your model and
all of your data to double-precision.  Round-off error in your results,
even after accumulating somewhat, should be reduced such that
you have about eight or nine additional decimal digits of precision.
You comment that converting your “entire system to .double() …
seems to yield larger problems than without it.”  Is this after
training multiple iterations after your parameters may have started
wandering down different paths?  Or is this still the case after only
a single forward and/or backward pass through your system?
Best.
K. Frank