Hi kliu!



 kliu:

do you happen to know why BCEWithLogitsLoss is numerically more stable than applying Sigmoid and BCELoss separately? I’m assuming this might be an implementation detail?


Yes, it is an implementation detail.  Mathematically, the two are the
same.
The issue is that Sigmoid (in particular, the so-call logistic function)
uses exponential to map (-inf, inf) to (0.0, 1.0).  But then BCELoss
turns around and uses log() to map (0.0, 1.0) back to (-inf, inf).
Mathematically, this is fine, but numerically, using floating-point
arithmetic, the exponentials can start to saturate, leading to loss
of precision, and can underflow to 0.0 and overflow to inf, leading
to infs and nans in your loss function and backpropagation.
BCEWithLogitsLoss avoides this internally by rearranging the
computation.  (Note that pytorch provides a LogSigmoid function
that does the analogous  computation internally.)
A similar issue arises when feeding the results of Softmax to a
plain cross-entropy loss.  Pytorch doesn’t even offer a plain
cross-entropy function.  Instead, pytorch’s CrossEntropyLoss
requires logits as its inputs, and, in effect, applies the Softmax
internally.
Best.
K. Frank