if a feature appears as the most important among a variety of models, does that make this feature empirically important? Lets say that I train a variety of different models - SVM, XGBoost, LogisticRegression, Random Forest, KNeighbors, etc. I then take the permutation importance of each model, and find that Column12 is the most important feature among all the models.
Assuming that this is not a mistake (i.e. hyperparams were tuned and retrained, cross-validation was used, strongly correlated columns were eliminated, etc...), does that mean that Column12 has explicit predictive power as a feature? Why or why not?

Does this mean Column12 important to each model? Yes. Does this Column12 mean is important to the outcome (assuming the outcome variable is the same in each model)? Not necessarily.
Permutation importance measures the contribution of altering the feature to the degradation of a model performance metric (typically the mean squared error). Just as its agnostic on the technique that generated the underlying model, it doesn't say anything about importance to the outcome.
