How is Kolmogorov Smirnov a good measure of classifier performance? KS tells how different two distributions are from each other. Assume I have a classifier model that has completely misunderstood the data and predicts a 0 for every 1 and a 1 for every 0. Assume the probability distributions are really far apart.
Wouldn't this model have a horrible accuracy and a really good KS, since KS doesn't measure accuracy but only distribution difference? If I measure my model per KS it says I will have an excellent model when it is garbage at correctly predicting the output.
What am I not understanding in regards to using KS to evaluate classifier performance?
EDIT: I am asking because I've encountered KS as measure of model performance in online articles and professional settings

It is not.
First, KS only assesses the maximum vertical distance between CDFs. Consequently, it will not be sensitive to differences elsewhere that a metric like Brier score or log loss will detect. Whenever either of those metrics encounter a prediction that differs from the observed value, there is a penalty.
Second, as you have noted, the KS test misses reverse calibration. If my predictions are consistently the opposite of what they should be, there are ways to handle that, but I need to know, not be deceived into thinking my predictions are good.
A combination of a poor score on a metric that catches reverse calibration and a high score on KS (so a low p-value) could be a signal that there is reverse calibration. I suspect, however, there to be considerably better alternatives.
