Hi Sam!



 shartzog:

I’d like to be able to produce an estimation of the likelihood of “HomeRecovery” for an individual sample.


The more I think about it, I do believe that your proposal to train on
a “HomeRec” label is the way to go.
A disclaimer:  Because your data is not labeled with probabilities,
you’re not training your model to predict probabilities.  Consider a
case where you overfit your model on your training data. You’ve
trained it to predict 0s and 1s, so you get mostly predictions along
the lines of  0.01 and 0.99 with a very low loss and very high
accuracy on your training set.  But let’s say that you get a useful,
but middling accuracy of 75% on your validation set.  Your model
(because it was overfit) has high confidence in its predictions (close
to 0.0 or 1.0), but these are not related to non-training-set
probabilities.
Back to your question:
You could add the additional “HomeRec” label to your multi-label
classification problem, train that, and interpret your “HomeRec”
prediction values (converted from logits to probabilities, as appropriate,
depending on the output of your model) as the probability of a single
sample being “HomeRec.”  To tie this back to the earlier discussion,
we can think of the model “learning” how the correlated “Hospitalized”
and “Deceased” probabilities should be combined to produce the
“HomeRec” probability.
Note, when training such a network, you should probably stop the
training when your training loss and validation loss first start to diverge
even if further training were to produce a model whose validation
and/or test loss and accuracy were better.  This is because you’re
trying to make your predictions best reflect their own uncertainty,
rather than make them as good as possible, but without a good
measure of their uncertainty.
Here’s a couple more things to consider:
If you want to focus narrowly on predicting the “HomeRec” probability,
then perhaps you should train a pure “HomeRec” vs. everything else
binary classifier.  Then the classifier won’t make compromises to
get the other classes right at the expense of not doing as well on
“HomeRec.”  But you could argue to the contrary that by not training
on the other classes you’re giving your training procedure less
information.  Perhaps learning the other classes helps your model
learn relevant “features” that help it do a better on “HomeRec.”
Another point (that I’ll outline in the context of a pure “HomeRec” binary
classifier):  If your dataset is unbalanced, it might be best to train your
classifier with weighted sampling or class weights in the loss function
so that its predictions won’t be biased in favor of the class that occurs
more frequently in your training set.  But, coming back to the issue of
predicting probabilities, if the fraction of “HomeRec” vs.“Not HomeRec”
in your training set is representative of the “HomeRec” fraction in the
“real-world” data to which you’ll be applying your model, then perhaps
you do want your predictions to be so biased.  Consider the extreme
case where your model gives the same prediction value, regardless
of its input:  If your real-world and training data both consist of 30%
“HomeRec” samples, then you would want that single prediction value
to be 0.30, rather than the “unbiased” 0.50.
Good luck.
K. Frank