Hi Py!



 pytorcher:

my custom functions were using .data in 0.3 and 1.3 versions.


My conclusion is that your use of .data() is the cause of “does
not work at all in 1.3.1.”


my custom functions were not needed to replicate this problem.


Note, what you show below does not replicate the "does not work at
all " problem – it replicates the “agrees up to expected floating-point
round-off” so-called problem.


Example output:

is same?
-0.0011598279234021902
-0.0011598279234021902
-0.0011598295532166958



I assume that this is running on 1.3.1 with (the default) single
precision (.float()).  Note, that your third result (from autograd)
agrees up to single-precision round-off with the first two.  This is
really the best you can expect.


Edit: with double precision it does agree much more but still not the same.

is same?
-0.00020605623917282047
-0.00020605623917282047
-0.00020605623917282427



Now I assume this is 1.3.1 with double precision (.double()).
Your results still agree up round-off error, but this time up to the
more accurate double precision.


Is this just the best that can be expected


Yes, this is the best you can expect.  It should not be viewed as
something being wrong, or any kind of issue (other than that’s
how floating-point arithmetic works).


because of what autograd is doing under the hood?


I wouldn’t say that this is “because of … autograd.”  You do the
gradient calculation twice – once your way, and once autograd’s
way.  Neither one is more right or wrong than the other.  They
just happen to perform certain calculations in different orders.
The different orders are mathematically equivalent, but differ
when using floating-point arithmetic.  So you get differences at
the level of floating-point round-off.
But, again, it’s not autograd, per se.  You could, yourself, calculate
the gradient twice, but purposely reorder some of the calculations
in a mathematically equivalent way, and you would see similar
differences.
Best.
K. Frank