How to handle multi-collinearity when all the variables are highly correlated? In my dataset, all the variables are highly correlated (correlation coefficient > 0.95). However, the correlation with the dependent variables is very low (<0.35). I checked the variation inflation factor (VIF), but it's 'inf' for some variables and very high (around 6 figures) for other variables. So cannot filter with the threshold of 10 as usual but setting a threshold around 1000 does not seem to be correct either. The response is a binary outcome. I'm trying to build a logistic regression model but due to high collinearity coefficients are not interpretable. How can I handle multi-collinearity in this case?
EDIT: I want to use the model to check what variables are significantly associated with the outcome and also to predict the outcome. The training set size is 128 (after splitting into train-test) and there are 96 variables in total. Data is highly imbalanced.

The correlations and VIFs are telling you the story. All of your features are extremely related to each other, so any notion of increasing one while holding others constant is nonsense, and I see no hope for untangling your features.
It’s nice to be able to have a clean interpretation, but the universe does not owe you one.
Further, you have way too many variables for your sample size. Even if you combined your training and testing data, you would have more variables than instances of the minority class. I fear that whoever tasked you with doing this has, probably unknowingly, sent you on a snipe hunt.
Finally, while VIF gives some idea of how features are related, in a logistic regression, the usual VIF does not have its literal interpretation as the factor by which the coefficient variance is inflated. When you work on a more reasonable logistic regression problem, you might want to consider generalizations of the usual VIF.
