Is this a correct interpretation of percent importance? In this scenario, the end goal is to determine which columns are most important. I use this term very very very loosely, because I know that interpretability is never straightforward but I just want to get a general idea about a ranking for column importance.
Say that I have 5+ different tree-based models that I train on the data. I then gather the percent importance for each of the features, for each of the models. At this point I have a dataset of however many features I started off with, multiplied by however many models I am using, and each of the features' percent importances per model.
I then calculate how many times each feature appear in the top 10 by percent importance, per model. So at this point, I have calculated something like

*

*column1 is in the top 10 by percent importance for 8/10 models

*

*I can conclude that since column1 is relatively important for a lot of the model, it is a generally important feature



*column2 is in the top 10 by percent importance for only 2/10 models, and column3 is never in the top 10 for any models

*

*I can conclude that column2 and column3 are relatively unimportant



Does this approach make sense/is my interpretation correct?
Notes:

*

*I am only using tree-based models which have percent importance attributes to them (RandomForest, CatBoost, ExtraTrees, etc...)


In contrast to frank, I think this method is potentially useful. I think we can reasonably disagree here. I am less concerned with what frank sees as a problem: since you already look at multiple models, you are already trying to account for the fact that variable importance is (indeed) only measured within the context of a specific model.
What I would be a bit careful about is any correlation between models. If your models are highly correlated (say, different Bayesian models with the same predictors and only small differences in priors), then they will likely output the same predictors as "important" - but this agreement does not mean much.
So: if you do this, try to ensure your models are as dissimilar as possible. (In which case the question arises whether the importance percentage measures are still sufficiently comparable.)
