Hi Stark!



 stark:

Please share your thoughts on having a soft target in a soft loss function.


This is a perfectly reasonable thing to do. BCEWithLogitsLoss (as
well as BCELoss) supports soft targets – you just pass them in as
the target.  You can certainly contemplate using soft targets with
other “soft” loss functions, with the details depending on the specific
loss function.
(As an aside, it also makes sense to use soft labels with cross-entropy.
Pytorch’s CrossEntropyLoss doesn’t support soft labels, but it is
straightforward to write your own soft-label version of cross-entropy.)


Could it be a better option?


Probably not (but it depends).
Your targets are what you know about your training data.  If you
have (correct) hard class-“0” / class-“1” labels there is no benefit
in artificially fuzzing them into soft labels – you would just be throwing
away information.  On the other hand, if your labels are naturally
soft, you probably make things worse by making them artificially
hard.
To illustrate what I mean, consider training a binary cat-dog classifier.
If all of your training images are clear cut – “Yup, that’s a cat; yup
that’s a dog.” – then you should use hard labels because doing so
uses all of the information you have.
But if some of your images are truly uncertain – maybe they are blurry
or partially obscured – you should use soft labels, because that’s what
you actually know.
In practice, why might you have soft labels and how would you get
them?  You could, for example, have 100 people label each image.
If 93 people label a specific image as “dog” (and 3 as “cat”), you
would give that image of soft label of 0.93.  If 24 (of the 100) people
label an image as “dog,” you would give it a soft label of  0.24.
The fact that people don’t always agree about the images indicates
that the image labels are uncertain, that you might want to use soft
labels, and gives you a quantitative way to assign such soft labels.
Best.
K. Frank