Hi Py!



 pytorcher:

It gives the same up to 16 decimal places for the first few batches but then drift starts to accumulate.


This is strong evidence that your two versions are doing the same
thing, up to floating-point round-off.  You can’t expect any more
than this.
(As an aside, 16 decimal places is what you would expect for
double precision.  So you’re actually doing better than you would
expect for single precision.  This behavior is actually a bit odd,
but I can cheerfully concoct some plausible explanations for it.)


With double precision the drift started immediately.


Whether this is a concern depends on the details of what you
mean by this.  After a single forward or backward pass, do your
double-precision results agree to significantly less than 16 decimal
digits?  If not, then you’re getting as much as you have an right
to get.
Best.
K. Frank