Hi Simon!



 smonsays:

I have uploaded an instance of tensors for which the two losses diverge
# ...
print("BCE loss with logits: ", BCE1)  # -2.3662e+08
print("BCE loss with sigmoid: ", BCE2) # -379848.2500
print("Loss difference: ", BCE1-BCE2)  # -2.3624e+08

The x_recon values are all very small (around -1.0e+05). However, I am not sure why the version without logits behaves so differently in this regime. Do you have any insights in this regard?


First, unfortunately, I’m not able to load your sample data with
my creaking, wheezing, 0.3.0-version of pytorch.  If they’re not
too large, could you post them as text files?
You say, “The x_recon values are all very small (around -1.0e+05).”
You give a numerical value of -10,000.  I would call this “a rather
large negative number.”  (“Small,” to me, has the connotation of
“close to zero.”)
Anyway, let’s go with -10,000.  That is, your logits (inputs)
are rather large negative numbers.  So your probabilities
(sigmoid (logit)) are all (positive) numbers quite close to zero.
But (with 32-bit floating-point) they underflow and become
exactly zero.  (32-bit sigmoid underflows to zero somewhere
around sigmoid (-90.0).)
Given this, I’m surprised you’re not getting NaNs (from the
log (0.0) inside of binary_cross_entropy()).
Anyway, could you tell us the shape of x_recon and x_view,
as well as the (algebraic) minima and maxima of x_recon
and x_view?
Assuming that your x_recon really have become something
like -10,000, it’s not surprising that you’re getting weird results
(at least for plain bce) – you’ve long since passed into the
range where sigmoid() underflows.  (This still doesn’t explain
why you’re getting seeming good results with plain bce, but
things break down with bce_with_logits.)
Best.
K. Frank