Do you know any technique for checking if a machine learning regression model gives different performance for different values of features? Let’s say I have fitted a regression model on a large dataset, and I obtain an estimation performance metrics (R-squared, RMSE) for the whole dataset, via cross validation or via a test set. However, this estimation is relative to the whole dataset, with no difference in different parts of the feature space. It would be possible that the model perform well for certain values of the features, and bad for certain other values.
How can I detect that? Is there any technique?
The only thing I can imagine is to build several test sets containing data with different feature values. But that is gonna be very complicated and arbitrary if I have a large number of features.
Do you have any idea?
Thanks a lot in advance,
All the best
Davide

your crossvalidation estimates of each point are valid out of sample estimates. so with any summable metric eg  logloss,MSE, but not RMSE,  you can just breakdown the total loss along each feature.
eg if one of the features is age_bracket
then overall MSE = weighted mean of Squared error in each age_group
so you can analyse your loss with a pivot table of dimensions and squared error on each datapoint
so if your crossvalidation mse is 85, you might break it down by age and see that the 40 -80 year olds have the largest  source of error.




age
proportion
Squared error




20-30
50%
20


30-40
25%
100


40-80
25%
200


Total
100%
.5x20 + .25x100 + .25x50=85



