How many ways are there to check model overfitting?
I am running xgboost on a regression classification problem where the model is predicting a score of how likely a gene is to cause a disease from 0-1.
I try to avoid overfitting in all the ways I can think of and the mean output of nested cross-validation is r2 0.88, I'm not sure if I can trust this or if there are other ways I can see if this is overfitting. The output r2 on just training and testing non-nested is: Train r2: 0.971 Test r2: 0.868.
So far I: - Remove features with a correlation >0.9 and remove any features with >50% missing data (this is hard to strengthen, a lot of genetic features simply have missing data for a lot of under studied genes in biology) - Have no imputation to avoid imputation bias, and since xgboost accepts missing data. - Scale features with MinMaxScaler() in scikit-learn - recommended as a good starting point and most features don't have a normal distribution - Compare 2 feature selection methods (one using features xgboost deems important from SHAP values and one using Boruta, both give 0.87-0.88 r2 on average of the 10 nested CV k-folds and only remove 3-4 out of 57 features) - Use nested kfold cross validation with 10 kfolds
The only other area I'm aware of that I haven't really explored is projection techniques. I am not sure which method would be best for this (my features are all numeric but mixed continuous or discrete data types) such as between UMAP, PCA or partial least squares.
Are there any other ways I can investigate overfitting? I have a biology background so any resources on this would be useful and any help appreciated.
I have also more manually removed some minority example genes before training (e.g. 392
removed training genes with a 0.9 score which make up only about 1/8 of the training dataset) to give the trained model to predict and view how the model generalises to this 'new' hard to predict genes - gives them a 0.6-0.7 score when they are actually 0.9:
``` y_pred =<PHONE_NUMBER>, 0.709764, <PHONE_NUMBER>]
y_true = [0.9, 0.9, 0.9]
r2_score(y_true, y_pred) #outputs 0.0
```
10-fold nested cv r2 results per fold: 'test_r2': array([0.8484691 , <PHONE_NUMBER>, <PHONE_NUMBER>, <PHONE_NUMBER>, <PHONE_NUMBER>, <PHONE_NUMBER>, <PHONE_NUMBER>, <PHONE_NUMBER> , <PHONE_NUMBER>, <PHONE_NUMBER>]
Edit:
A few other things I've tried:
* I think I've misused classification here (and removed tag accordingly), I use regression models and I don't have labels and only continuous scores so I don't get true positives, false positives etc. to be able to do ROC. I'm not sure what other metrics are good or better than R2 for regression that I can use.
* I have tried applying imputation to compare other models (random forest, SVM, and logistic rgeression with elasticnet or lasso), all models perform notably lower than gradient boosting (0.59 average nested r2 is the highest with random forest) - but I was originally concerned with biased data from imputation, is imputation worth doing to counteract overfitting?
* I use GridSearch in scikit-learn for all my models with nested cross-validation, I should have included this information originally as I have been trying to always do this.
I have a biology background, so not sure about best practices for machine learning, but from this I'm suspecting random forest is better and I should be trying to do a better parameter tuning than I currently do for it, and trusting that model's result on nested CV. Is this the best approach?
Also not sure if how I tune my random forest is reasonable, currently I use: rfr = RandomForestRegressor(random_state=seed) rfr_params={'n_estimators':[100, 500, 1000], 'min_samples_split': [50, 100], 'min_samples_leaf': [50, 100],} 392