Hi Sergey!



 my3bikaht:

But by applying log, your new minimum will be achieved if ce loss = 1/N, bcs log(1)=0


To avoid any confusion, let’s be clear that this is not correct.
It is true that many loss functions have zero as their minimum, but
gradient descent and the pytorch optimizers don’t care whether this
is true or not; they simply push the loss function to smaller – and
potentially negative – values.
log() is a so-called monotone function so log (ce_loss) achieves
its minimum exactly where ce_loss achieves its minimum.  ce_loss
ranges from 0.0 to inf and, correspondingly, log (ce_loss) ranges
from -inf to inf.  When your predictions are “perfect,” ce_loss is 0.0,
its minimum, and log (ce_loss) is -inf, its minimum.
Whether you train with ce_loss or log (ce_loss), the optimization
drives both towards their corresponding minima, namely 0.0 for  ce_loss
and -inf for log (ce_loss).  Nothing special happens when ce_loss
becomes 1/N (or 1 or any other value other than zero), and the fact that
log (ce_loss) happens to be 0.0 when ce_loss = 1/N is irrelevant;
the optimization simply continues to drive log (ce_loss) to algebraically
smaller values, and thus to negative values on down towards -inf.
(Consider what would happen if one were to use log (ce_loss) + 17.2
or log (ce_loss) - 105.9 as the loss function for training.  Would the
gradients change?  Would the value 17.2 or 105.9 have any effect on the
result of training?)
Best.
K. Frank