Hi Py!



 pytorcher:

But then on the very first forward pass I get losses of 1.9986734118816107 and 1.9986734009655582.


If I understand correctly, this happens when you are running everything
(as far as you know) in double precision.
This is odd.  Your two results agree up to single precision, so it’s not
as if one version is completely broken.  It’s as if one version does
something only in single precision, even though it should be in double
precision.


When I was only using float precision things did always stay exactly the same for a few batches before the drift started.


Do I understand correctly (based on your earlier post) that this means
that the two versions agree (for a few batches) up to double precision,
even though you are running in single precision?
This is also odd (although not “wrong” or impossible).  Am I right
that you are saying that when running in single precision, the two
versions agree up to double precision, but when running in double
precision, the two versions only agree up to single precision (at least
for the first batch)?


But I am only debugging with this very simple model,


Could you post a simple, complete, runnable script that illustrates
this issue, together with its output?


The evidence something is wrong is that with the real model everything works reliably in pytorch 0.3 and does not work at all in 1.3.1.


What does “does not work at all in 1.3.1” mean?  It sounds like (in
single precision) 1.3.1 agrees with 0.3 up to double precision.  So
that sounds like it’s working.  (And in double precision, they agree
up to single precision, so it’s more-or-less working.)
Also, could you tell us the exact version of your 0.3.0 version?
(You can run print (torch.__version__).)
Best.
K. Frank