Usefulness of KS tests and other similar distribution comparing tests I am working on a machine learning binary classification problem.
I have an outcome variable status called as loan paid and loan default.
So, I have the loan amount data for each of the loans.
Now, I would like to know whether the loan amount borrowed is significantly different for loan paid and loan default applicants.
So, I tried KS-test to compare the loan amount for these two groups.
Am I right to do this way?
data1 = loan_df[(loan_df['status']=='default')]['loan_amount'].reset_index(drop=True)
data2 = loan_df[(loan_df['status']=='paid')]['loan_amount'].reset_index(drop=True)
from scipy.stats import ks_2samp

#perform Kolmogorov-Smirnov test
ks_2samp(data1, data2)

I get a result as below

KstestResult(statistic=0.504473708788643,
pvalue=5.551115123125783e-16)

My questions are as follows
a) What is the usefulness of these tests? Meaning, if they are significantly different from each other, do they help in predictive power?
b) what kind of inference can be made from such tests?
c) Is there any other methods available to compare two distributions? So, I can double confirm/cross-verify that it is indeed different

This is one of those ideas that starts out sounding great but winds up being less helpful than one might hope.
For instance, just because KS (or a similar test) says that the feature has a different distribution for each outcome category doesn’t mean that every model will care. If the distributions only differ in variance, for instance, a linear predictor in a logistic regression will not catch that. Consequently, your feature will wind up being a dud that lacks predictive ability, despite a difference in variance between your outcome.
Consequently, distribution differences need not reveal important predictors.
On the other hand, even if univariate distributions are identical, multivariate distributions might not be. For instance, imagine a scatterplot of your points that form an X shape, where the “slash” portion is one group and the “backslash” is another. Those univariate marginal distributions are the same for each group, yet those predictors have considerable discriminative ability. Not every model will be able to use the correlation, but some will.
Consequently, just because you get a result that the distributions are similar or even the same does not mean that a feature is useless.
Overall, we cannot conclude much from this kind of screening.
EDIT
Let's look at some simulations.
1)
set.seed(2022)
N <- 1000
x0 <- rnorm(N, 0, 1)
x1 <- rnorm(N, 0, 2)
x <- c(x0, x1)
y <- c(rep(0, N), rep(1, N))
ks.test(x0, x1)
L1 <- glm(y ~ I(x^1), family = binomial)
L2 <- glm(y ~ I(x^2), family = binomial)
summary(L1)
summary(L2)
ks.test(x0, x1)

In this simulation, $Y=0$ and $Y=1$ have different distributions on the $x$ feature. However, since the difference is in the variance, a logistic regression that only uses a linear predictor does not care. We have to know to analyze the quadratic term to get a significant predictor. This means that we either have to know what kind of difference there is between the distributions (which the KS test does not provide) or allow for some kind of flexible model that will figure it out, as Frank Harrell mentions in the comments.
Consequently, a significant result from the KS test does not help us determine the features to include in the model.
2)
library(MASS)
set.seed(2022)
N <- 1000
x0 <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, 0.9, 0.9, 1), 2, 2))
x1 <- MASS::mvrnorm(N, c(0, 0), matrix(c(1, -0.9, -0.9, 1), 2, 2))
x <- rbind(x0, x1)
y <- c(rep(0, N), rep(1, N))
ks.test(x0, x1)
L0 <- glm(y ~ x[, 1] + x[, 2], family = binomial)
L1 <- glm(y ~ x[, 1]*x[, 2], family = binomial)
summary(L0)
summary(L1)
ks.test(x0[, 1], x1[, 1])
ks.test(x0[, 2], x1[, 2])

Here, the KS test says that both features have the same distribution for both levels of the response. However, the joint distribution is dramatically different, due to the difference in correlation (one strongly positive, one strongly negative). Consequently, a model that has an interaction term (such as L1) or that discovers such interactions (e.g., neural networks, SVMs with certain kernels) winds up screaming that the features have discriminative ability.
Consequently, an insignificant result from the KS test does not help us determine which features should be excluded from the model.
