Hi roxor!



 roxor:

# input values are logits
# 4 possible classes and 2 samples 
inputBad = torch.tensor([[3,5,2.5,6], [7,5.5,2,0]])
inputGood = torch.tensor([[0,0,0,1], [1,0,0,0]], dtype=torch.float)



Your inputBad and inputGood don’t represent the predictions you
think they do.  As your comment notes, they are logits.  To make them
a little easier to understand, let’s turn them into probabilities with
softmax():
>>> torch.nn.functional.softmax (inputBad, dim = 1)
tensor([[3.4387e-02, 2.5408e-01, 2.0857e-02, 6.9067e-01],
        [8.1249e-01, 1.8129e-01, 5.4745e-03, 7.4090e-04]])
>>> torch.nn.functional.softmax (inputGood, dim = 1)
tensor([[0.1749, 0.1749, 0.1749, 0.4754],
        [0.4754, 0.1749, 0.1749, 0.1749]])

First note that inputGood assigns your ground-truth label (class-4
and class-1 for your two samples, respectively) of 47.5% for both
samples.  Your ground-truth label does have the highest probability,
but it does not have a probability of 100%.
To repeat this, the input vector, [0, 0, 0, 1], does not mean a
probability of 100% for class-4 – it is a vector of logits that translates
to a probability of 47.5% for class-4.  So it is not a pure class-4
prediction.
Second, inputBad translates to ground-truth probabilities of 69.1%
and 81.2% for the first and second samples, respectively.  So
inputBad actually gives stronger predictions for the ground truth
than does inputGood.


penalize_target = torch.tensor([[5 , 8 , 2 , 0], [0 , 1 , 4 , 5]], dtype=torch.float)
target = torch.nn.functional.softmax(penalize_target, dim = 1)



Now let’s look at penalize_target:
>>> torch.nn.functional.softmax(penalize_target, dim = 1)
tensor([[4.7299e-02, 9.5003e-01, 2.3549e-03, 3.1870e-04],
        [4.8372e-03, 1.3149e-02, 2.6410e-01, 7.1791e-01]])

In both cases your target probability is smallest for your ground-truth
entry (class-4 and class-1, respectively).  The problem is that a small
number in your distance matrix (small is “good,” and zero is “perfect”)
translates into a small probability – backwards of what you want.
A couple of comments:
First, although you could reverse-engineer soft-cross-entropy to give
you a distance-matrix-like loss function by appropriately adjusting
your soft targets, you don’t really want a distance-matrix loss.  You
really do want something like cross-entropy, with its logarithmic
divergences.  In practice, it seems to train better.
Take a look at the numerical details of the class-4 / class-9 example
in the post I linked to.  That shows concretely how to penalize certain
misclassifications more than others.
Best.
K. Frank