How to get the probability a prediction is correct from a binary classifier I have an image binary classifier that where class a = 0 and class b = 1
When I receive a prediction of a single image, is working out the probability that the prediction is correct as simple as:
a:  0 = 100%, 0.25 = 50%, 0.5  = 0%
b:  1 = 100%, 0.75 = 50%, 0.5  = 0%
?

$$
P(Correct\vert Category=K) = \dfrac{
P(Category=K\vert Correct)P(Correct)
}{
P(Category=K)
}
$$
$P(Correct)$ is the accuracy. $P(Category=K)$ is how many times you predict category $K$ out of the total number of predictions you make.
For $P(Category=K\vert Correct)$, restrict your predictions to the correct predictions. Then look at what fraction of those are predicted to be $K$.
Unfortunately, you need the true labels to calculate $P(Category=K\vert Correct)$, so you cannot do this in the setting where you're making predictions on data where you truly do not know the labels (e.g., Siri or Alexa doing speech recognition on sentences that have never before been spoken). When you do have the labels (such as in a test set), however, this is related to the positive and negative predictive values, which tell you the probability of a case being positive or negative ($1$ or $0$ in standard categorical encoding) given a prediction of positive or negative, respectively.
But all of this assumes that you're making a categorical prediction. The neural network models you're using return probability values, not categories.
