I think you&#x27;re describing accuracy, as in prediction accuracy.<p>Confidence is simply how much confidence you have that the predicted class is correct.<p>This is often important because in a real system you often want to know whether to accept the model&#x27;s prediction or pass the case on to a more accurate, more expensive model (sometimes a human).  You then can select a confidence threshold that gives an acceptable trade-off between false positives and false negatives (ie. choose an operating point on the ROC curve).  If your model just gives a yes&#x2F;no answer with no measure of confidence, or if you don&#x27;t know how to interpret the output as a confidence, then you don&#x27;t have sufficient information to do this.<p>Over-fitting is often part of the problem but in my experience things aren&#x27;t as black or white as you seem to imply.  There are things you can do to limit over-fitting but I don&#x27;t know of any simple test that will give you a 0 if you&#x27;re over-fitting and 1 if you&#x27;re not.