Hi Simon!



 smonsays:

Both have dimensions torch.Size([100, 784]). The first dimension is the batch size, the second is the number of pixels in a MNIST training image. The extrema are:
torch.min(x_recon) # -16971.4434
torch.max(x_recon) # 15807.5469
torch.min(x_view) # -0.4242
torch.max(x_view) # 2.8215)

So it turns out there are also very big entries in x_recon that saturate the sigmoid towards 1.


Okay, this makes more sense.
The input and target passed to binary_cross_entropy()
are both supposed to be probabilities, that is, numbers between
0 and 1 (with singularities occurring at 0 and 1).
Your target contains negative numbers, which are not valid
probabilities.
(Because you pass your x_recon through sigmoid(), your input
will always contain valid probabilities, although they will sometimes
saturate at the singular 0 and 1.)
So my guess is that your bogus target values are causing your
training to drive your inputs to the large values that saturate
sigmoid().  (Why you’re not getting NaNs, I don’t know.)
Given your bogus target, I’m not surprised that you’re getting
weird results.  (I can come up with plausible speculations about
why plain bce and bce_with_logits differ, but that’s not really the
point.)
Figure out how to fix your inputs to binary_cross_entropy(),
specifically x_view, and see if that cleans things up, or at
least improves the situation.


I created a demo with minimal changes to the original example that reproduces the odd behaviour here.


(I haven’t tried to run your demo, because, among other reasons,
it likely won’t run with my decrepit pytorch 0.3.0.)
Good luck.
K. Frank