KFrank:

You say, “The x_recon values are all very small (around -1.0e+05).”
You give a numerical value of -10,000. I would call this “a rather
large negative number.” (“Small,” to me, has the connotation of
“close to zero.”)


Ah sorry about that, you are of course right, the x_recon values are large negative numbers. Only the sigmoid is very small.



 KFrank:

Anyway, could you tell us the shape of x_recon and x_view ,
as well as the (algebraic) minima and maxima of x_recon
and x_view ?


Both have dimensions torch.Size([100, 784]). The first dimension is the batch size, the second is the number of pixels in a MNIST training image. The extrema are:
torch.min(x_recon) # -16971.4434
torch.max(x_recon) # 15807.5469
torch.min(x_view) # -0.4242
torch.max(x_view) # 2.8215)

So it turns out there are also very big entries in x_recon that saturate the sigmoid towards 1.



 smonsays:

In the mean time I’ll try to figure out the minimal changes to the official example that are necessary to reproduce the odd behaviour and post the code here.


I created a demo with minimal changes to the original example that reproduces the odd behaviour here 4. The relevant change was the normalization of the data to zero mean and unit variance (i.e. transform = transforms.Normalize((0.1307,), (0.3081,))).