Hi Chinmay!



 chinmay5:

I want to create a custom loss function for multi-label classification. The idea is to weigh the positive and negative labels differently.


Do be aware that pytorch’s BCEWithLogitsLoss supports a
pos_weight constructor argument that will do what you want.
So unless this is a learning exercise, you should simply use
BCEWithLogitsLoss.


        y_hat = torch.sigmoid(input + 1e-8)



The 1e-8 doesn’t do anything useful here.  sigmoid() is very
well behaved (and equal to 0.5) when its argument is equal to
zero.  So there’s no need or benefit in trying to move the argument
a little bit away from zero.  (Furthermore, -1.e-8 is a perfectly
valid logit and argument to your loss function and your “fix” just
moves it to zero – not that anything bad happens at zero.)


        loss = -1.0 * (self.pos_weight * target * torch.log(y_hat + 1e-6) + self.neg_weight * (1 - target) * torch.log(1 - y_hat + 1e-6))



Here you apply log() to sigmoid().  This is a source of numerical
instability.  You should use the log-sum-exp 2 “trick” to compute
log (sigmoid()).  (This is what pytorch’s BCEWithLogitsLoss
does internally.)


I started getting nan values
…
Next, I again saw getting inf


With the code you posted, I don’t see why you would be getting
nans or infs.  The 1.e-6 that you add to your log() functions
should protect against that.
Best.
K. Frank