Hi David!



 davidsriker:

class WeightedBCE(nn.Module):
    def forward(self, y_hat, t):
        loss = 0.
        for label in range(self.pos_w.shape[0]):
            pos_loss = -1. * torch.mean(t[:, label] * self.pos_w[label] * torch.log(y_hat[:, label] + self.eps))
            neg_loss = -1. * torch.mean((1. - t[:, label]) * self.neg_w[label] * torch.log(1. - y_hat[:, label] + self.eps))
            loss += pos_loss + neg_loss

        return loss



First, as an aside:  I haven’t looked closely at your WeightedBCE code, but I
don’t see anything mathematically wrong with it.  However, the for-loop over
label will slow things down in comparison with pytorch’s built-in BCELoss
(and the preferred BCEWithLogitsLoss).


Where neg_w is:
tensor([0.2246, 0.0539, 0.0907, 0.0448, 0.2572, 0.0486, 0.0319, 0.0044, 0.3841,
        0.1113, 0.1222, 0.0656, 0.0283, 0.1016])

and pos_w is:
tensor([0.7754, 0.9461, 0.9093, 0.9552, 0.7428, 0.9514, 0.9681, 0.9956, 0.6159,
        0.8887, 0.8778, 0.9344, 0.9717, 0.8984])



Pytorch’s built-in BCEWithLogitsLoss supports positive-sample weighting
with its pos_weight constructor argument.  Note that, for whatever reason,
BCELoss does not offer this pos_weight feature.
BCEWithLogitsLoss does not have a corresponding neg_weight option,
but its weight constructor argument gives overall class weights.
I haven’t double-checked the math, but I believe that for you to reproduce your
weighting scheme with BCEWithLogitsLoss, you would want to construct your
loss-function object as follows:
loss_fn = torch.nn.BCEWithLogitsLoss (weight = neg_w, pos_weight = pos_w / neg_w)



My model output, is after sigmoid layer, but for some reason I can see any improvement in the custom hamming score during training:


It does look like your loss may be drifting down, although I agree that I do
not see any improvement in your “Hamming Score.”  However, you’ve only
trained for a single epoch, so I wouldn’t be too hasty about drawing any
conclusions.  Also, some batches can be harder or easier than others (and
differently so for your loss vs. other figures of merit), so batch-to-batch
variance may mask a systematic trend.
You should certainly experiment with learning rate and momentum, and
possibly try other optimizers such as Adam if you are currently using SGD.


I was thinking of using BCEWithLogitsLoss and remove the sigmoid layer from the model


You should certainly prefer BCEWithLogitsLoss over BCELoss with sigmoid
for numerical-stability reasons.  However, unless you are hitting issues with
numerical stability, the two will be the same, so switching won’t improve the
results of your training.  (But you should switch to BCEWithLogitsLoss anyway
to guard against future stability issues.)
Best.
K. Frank