Hello Rui An!



 ruian1:

I was using BCEWithLogintsLoss to train a multi-label network and getting negative loss as shown below.
…
criterion = nn.BCEWithLogitsLoss()
a = torch.tensor([[1., 1., 1., 0., 0.]])
b = torch.tensor([[0., 0.0011122, 8.9638, 0., 0.]])
c = nn.Sigmoid()(b)
print(criterion(a,b))
print(criterion(a,c))

"tensor(-0.7278)"
"tensor(0.6652)"



It appears that you have switched the order of your inputs to
BCEWithLogitsLoss.
BCEWithLogitsLoss (like
binary_cross_entropy_with_logits()) expects to be
called with predictions that are logits (-infinity to infinity) and
targets that are probabilities (0 to 1), in that order.
Your  a are legitimate probabilities, so they are your targets, and
your b are legitimate logits, so they are your predictions.  Your
call should therefore be:
print(criterion(b,a))

(In your version of the call your second argument, b, is out of
range (not 0 to 1), so the call returns the invalid negative result.
c, however, being the result of Sigmoid, is in (0, 1), so the
result of the call is indeed positive.)


Should line 598 of pytorch/torch/nn/moduels/loss.py be changed as below  ?
        return F.binary_cross_entropy_with_logits(nn.Sigmoid()(input), target,



No.  binary_cross_entropy_with_logits() has built
into it (implicitly, in effect) the sigmoid function (just like
BCEWithLogitsLoss), so you want to pass logits as the first
argument (input), not probabilities (nn.Sigmoid()(input)).
You shouldn’t expect to see an explicit call to Sigmoid in the
source code for BCEWithLogitsLoss.  In effect Sigmoid is
applied to input, but it’s implicit (using the “log-sum-exp trick”).
Good luck.
K. Frank