Feature importance with categorical data I am facing a problem with my PhD Thesis in Archaeology. After analysing the percentages of an animal (variable: Observation) by grouping them by each variable in the following columns (e.g., Median % of Cattle in Urban sites), I am interested in knowing which of these features is more important, rather than simply exploring different distributions. The problem is that all the variables (except the observation) are categorical, so I do not know precisely how to proceed. The question I am interested in answering is: 'Which is the most important factor in choosing this animal? Region, Geography, site type?'
I started reading a bit about decision trees, random forests, etc. but I am unsure whether I can use these types of analyses or how should I reshape my data in order to use them.
Any advice would be appreciated, I know this is probably a vague question, but unfortunately I am not a statistician and I am a bit stuck with this issue.
Thank you in advance!




Observation
Region
Geography_Type
Site_Type




40
North
Coast
Urban


25
South
Mountain
Farm


...
...
...
...




All feature importance measures come down to assessing how much the classification degrades if a particular predictor is either completely removed from the model, or randomly permuted. If a model that removes (or randomly permutes) a particular predictor yields predictions that are as good as a model that retains that predictor, then the predictor can't be that important. The difference in predictive performance is then typically normalized in some way.
Thus, a measure of feature importance really comes down to choosing an evaluation metric (if you predict numerical observations, this could be the Mean Squared Error, or in other cases, it could be Gini impurity or similar), deciding whether to assess this in-sample, out-of-bag or with a holdout sample, and then running multiple models, removing or permuting predictors.
Some of these approaches are already provided in standard tools, e.g., in Random Forests (as in the eponymous R package), where this is a very simple and natural extension of the core functionality. In other cases (e.g., OLS), you can do the same thing, but you may need to code it yourself.
In any case, there is really nothing particular about categorical predictors here, beyond having a model that can deal with them.
