Hi Py!



 pytorcher:

Does not work at all in 1.3.1 means that the built in pytorch functions work properly and and it will converge to a slightly different local minima without using my custom functions, but when it uses my custom functions it makes no difference at all while it used to in 0.3.
It looks like in 1.3.1 it actually is not giving consistent values to begin with.  When I run the same code initialized with the same values I can print the network weights, network output, loss value all to be exactly the same to 16 decimal points but then actual grads that get calculated only agree up to 7.


If I understand properly what you are saying, it sounds like you
have this issue only when you are using custom functions and
you are running on 1.3.1.
This suggests that there is a bug in 1.3.1 that is uncovered in
by your custom functions.  Or, I suppose, there could be a bug
in your custom functions that 0.3.0 is letting you get away with.
Could you post the code for your custom functions?


I can try to make a minimal example to show code if that would help.


A minimal example would be the best thing.  A complete, runnable
script (with all necessary imports, hard-coded or randomly generated
sample, and anything else needed so that it can be run just by
copy-pasting it from the forum) would be very helpful.


Here is what I’m doing for debugging now, the second grad value changes every time past the first 7 digits while the first manually calculated grad is identical every time to all digits printed:


This makes it sound like the bug – whether yours, or in 1.3.1 – is
associated with the .backward() of one of your custom functions.
A minimal example – that will presumably contain the code for
your custom functions – would be the best next step.  Please
also post the output produced by your minimal example, when
run on both 0.3.0 and 1.3.1.
Also, I’m not saying that you should necessarily do this, but have
you tried upgrading to pytorch 1.5 (or maybe 1.4) to see if this
issue still persists?
[Edit:]  One possibility:  Are you using .data() in your custom
functions.  .data() is deprecated since 0.4.0, and, although
still present, doesn’t work correctly for all use cases.  Please
see the Pytorch 0.4.0 Migration Guide 2 for details.
Best.
K. Frank