Hi  Subhankar!



 SubhankarHalder:

class DenseCrossEntropy(nn.Module):
...
    def forward(self, logits, labels):
        logits = logits.float()
        labels = labels.float()
        
        logprobs = F.log_softmax(logits, dim=-1)
        
        loss = -labels * logprobs
        loss = loss.sum(-1)

        return loss.mean()

But I do not understand what’s going on in this class.
…
I can see its calculating the log softmax but then why the multiplication with the labels? And why loss.sum(-1)?


Pytorch’s CrossEntropyLoss takes as its target (labels) a single
integer class label for each sample in the batch (i.e., a tensor of shape
[nBatch]).  That is, target == 2 means that class “2” is the right answer
with 100% certainty.
The notion of cross-entropy is more general, however, in that it compares
two probability distributions.
So you might wish to work with probabilistic targets. e.g. for nBatch = 1:
target = torch.FloatTensor ([[0.1, 0.2, 0.7]])

This means that this sample is in class “0” with probability 10%, in
class “1” with probability 20%, and in class “2” with probability 70%.
This more general version of cross-entropy (which Kaggle is calling
DenseCrossEntropy) is not supported directly in pytorch, hence the
need to implement it explicitly.  (But it’s not very hard, and you can
do it using pytorch tensor operations, so you get the full benefits of
autograd and gpu support.)


        loss = -labels * logprobs
        loss = loss.sum(-1)



The “multiplication with the labels” is multiplying the vector of
probabilistic labels (element-wise) with the vector of logprobs
obtained from the vector of logits (raw-score predictions).  The
sum is just summing these terms in the formula for cross-entropy
across classes (within a single sample, hence .sum (dim = -1)).
To recap one key point:  for DenseCrossEntropy, labels has
shape [nBatch, nClass], while for torch.nn.CrossEntropyLoss,
labels (target) has shape [nBatch].
You can see the formula for this general cross-entropy in Wikipedia’s
Cross entropy entry.
Compare this to the formula for pytorch’s CrossEntropyLoss 2 that uses
categorical class labels for its target.
(As an aside, you can use one-hot-encoded class labels as a special
case of probabilistic labels with DenseCrossEntropy.  These are
just labels that consist of all 0s (0% probability) for all of the classes
except for one 1 (100% probability) for the class that is being labelled
as “correct.”)
Best.
K. Frank