Interpretation and understanding of Random Forests when feature importance results vary with each run I have a question that I have been wondering about for a while now and have not been able to find any reasonable answers regarding it.
How does one interpret Random Forests and feature importance when the results vary each run?
Example:
Let's say that a person wants to use random forest regression on their clinical data. The person chooses the features they want to use and also specifies the parameters they want to use for their model. The person makes the mistake of not setting a seed when running his model and is left wondering why his results aren't reproducible. When he is reminded by his friend that he needs to set a seed in order to reproduce the results that he got when analyzing his clinical data, he asks himself how do I know these results are actually "sound" if I need to set a seed to reproduce them? The person's friend explains that the results produced are for the observations that were taken in that specific run and that the model is stochastic. His friend suggests that if he wants results with less variance, he should increase the number of trees that he uses for his model. When the person increases the number of trees in his model to a much larger number (from 200 to 1600 trees), he notices that his two most important features never varied in their ranking between each run. But the person has 15 other less important features that still vary with each run, even on a larger number of trees. The person is now wondering, how can he explain his results when his 15 other features are still varying with each run? When reporting the features used in his clinical data analysis, how can he explain the importance ranking of each feature? Should he explain that the features that do not vary in between runs at a large number of trees are the ones that are important?
I have been pondering these questions for some time now.
Any help at all would be greatly appreciated!

If I were this person, I would apply bootstrapping and obtain feature importance averages with deviations, then sort the features and interpret. This is what typically done to battle with stochasticity. In each bootstrap iteration, you should choose a different seed.
