Hi Amirali!



 NoobCoder:

I have used nn.BCELoss for the loss function. Based on the Pytorch Website, it is mentioned that the output of the loss will be clipped to -100 if it is smaller than this value.


Just to be clear, the BCELoss 16 documentation says that the individual
log() terms in BCELoss will be clamped to be >= -100, not the loss
itself.


My question is that what will happen in the backward? If the y is zero, because of the gradient it will be in denominator, and the gradient would be inf.


My understanding of this (but I haven’t checked the code) is that
pytorch uses the “clamped-log” version of BCELoss rather than the
“true” BCELoss, and that the gradient produced by backward()
is the gradient of this modified loss function.  That is, you modify
the loss function and then take the modified function’s gradient,
rather than modifying the loss function and its gradient separately.
(The derivative of the clamped-log term is well defined, and diverges
nowhere.  It is zero in the clamped regime, but this doesn’t cause
any divergence.)
(As an aside, the documentation uses x for the input (prediction)
and y for the target.  It is x that appears in the log() terms, so it is
log (x) that gets clamped.  backward() also computes the gradient
with respect to x (not y).)
Best.
K. Frank