How to compare two random forests in scikit-learn? With most learning algorithms, one can compare the models resulting from applying the algorithm on samples of data by the parameters of the models. For example, one can compare two logistic regression models by comparing the learned model parameters (I'm not referring to the hyperparameters here).
I would like to do that with two random forest models trained with scikit-learn's random forest algorithm. However, I do not see any properties or methods that give me access to parameters that can be used to draw a comparison.
How would you compare two random forest models?

*

*For the overall fit, any binary models can be compared, irrespective of how they were trained, using some statistics which are computed based on the model outcomes. For all these measures it is better to use probabilities rather than binary class predictions, i.e.  Prob( class = 1| predictors).

a) ROC-AUC. In R you can get ROC-AUC with confidence interval, so can do a statistical test if one model is >> other in how it separates two classes, in scikit I didnt check, but you can try bootstrap to get confidence intervals.
b) Second measure is R2 for binary outcomes - scaled Brier score, it is less common, but more powerful measure of the overall model fit. You can google it, but essentially it is just R2 for a regression, where predicted y is model's probabilities and outcome is 1/0 - true class.


*If you are interested in the impact factors of each of the risk factor, you can use variable importance impact as suggested above, or also use SHAP library. It plots the size and sign of each of the risk factors for each observation in the training set in one graph, you can compare these graphs for your random forests.
https://towardsdatascience.com/explain-any-models-with-the-shap-values-use-the-kernelexplainer-79de9464897a


*You can compute predicted odds ratio yourself, e.g. take a median observation with all risk factors set to means, compute predicted probability of class 1, then change a risk factor by 1sd and see the change in prediction. You can compute odds ratios for each risk factor and compare those between your random forests (or any other models). Ideally, you want to bootstrap to get confidence intervals for those ORs.   Note, that in logistic regression these ORs will be the same across all observations, not just median ones, in RF those can vary a lot depending on what values you set for other parameters - so here you may need to see ORs for what group of people (median or some others) you want to compute and compare between the models.  The advantage of computing OR by hand is that you can compute for the target group, and also change ORs to absolute risk change or any other outcome measure that you want to compare.
