Does it make sense to apply Bayesian formula on top on a classification problem output? In classification tasks we normally get a set of numbers that represent a probability distribution - they sum to 1. For further discussion, suppose we only have two classes: positive and negative. Suppose, we assign a score of $S$ to a positive sample, thus implying that the score of the negative sample is $1-S$.
My question is, does it makes sense to apply the Bayesian formula on top of this classification result? For example, does it make sense to do the following:
$P(positive|s\ge S) = \frac{P(s\ge S|positive)P(positive)}{P(s\ge S)}$.
In terms of components of the formula, we have the following:

*

*$P(s\ge S)$ can be estimated from our training data by just looking at the rate of producing a score at least as high as the score we observe.

*$P(s\ge S|positive)$ can also be estimated from the training set, I suppose? As in, when we deal with a positive sample, how often do we produce a score at least as high as the one we observe?

*Finally, $P(positive)$ is an interesting one for me. Firstly, we can get an estimation of how often we expect to see a positive sample and if for whatever reason, the expectation is being driven up, we can account for that by changing the term in the formula, thus driving the whole probability estimation up.

I understand that when we consider $s\ge S$, we actually deal with floating-point numbers, but we can round them to two decimals and pre-calculate those probabilities at 100 points between 0 and 1.
I understand that this particular approach will not work well for such applications as computer vision, but I believe there is a wide range of applications where the components of the formula can be estimated reasonably easy.
Does this approach make sense at all?
Appreciate your input!

Sure you can. Bayes theorem is a general probability theory theorem, it can be applied to any probabilities (or probability densities). Bayes theorem is useful because it lets us "reverse" the direction of conditional probabilities from $p(A|B)$ to $p(B|A)$, whatever they are.
As for your case, do you really need the Bayes theorem here? Using SQL pseudocode, $P(S\ge s | positive)$ is just
select sum(S >= s) / count(*)
from predictions
group by class = positive

so can be calculated directly, without the Bayes theorem.
Notice that your proposed metric is quite similar to sensitivity which is the fraction of the hard classifications as positive for the subset of items labeled as positive, so basically $P(S\ge threshold | positive)$.
