In what applications do we prefer Model Selection over Model Averaging? I'm wondering in what applications or scenarios (or in trying to answer what kind of questions), the researcher would prefer using Model Selection (such as AIC or BIC) over Model Averaging (such as Bayesian model averaging)?  One example: if my question at hand is what's the most appropriate model, then I must use Model Selection.  Another (not sure) example: I guess  if we want less bias, Model Selection is also preferred.       It would be great if you have other examples like these. Paper or book recommendations are also welcome.

Model specification is a better approach than either model selection (by which most people refer really to feature selection) or model averaging.  Here are some pros and cons.

*

*model specification requires much more up-front thinking with regard to nonlinearities, non-additivities (interactions), and model components to penalize but this thinking pays off in downstream computation and interpretation and results in accurate estimates of uncertainty.  One example of model specification would be inclusion of a sensible number of nonlinear basis functions to allow for flexible modeling without assuming linearity, and specifying skeptical prior distributions for non-additive effects in a Bayesian model (or parameters to penalize in a frequentist setting, guided by cross-validation to get penalized maximum likelihood estimation)

*model (feature) selection does not work as advertised as you will almost always be disappointed to find that the "found" features do not replicate in future samples, collinearities almost destroy the ability to select, and predictive performance is not as good as fitting full models with appropriate penalization (e.g., ridge regression typically predicts better than lasso or elastic net).  Feature selection results in an example model not the model.

*model averaging accomplishes the same thing as a carefully pre-specified very flexible single model, but with a lot more work and difficulty in interpretation.  That is, if the domain of models being averaged are from the same family.  Sometimes model averaging over different model families can take care of model family uncertainty though.  For example you may be unsure of the link function to use when Y is binary and you are developing a probability model.  Or in a time-to-event analysis you might ponder a proportional hazards model vs. an accelerated failure time model.  But judicious use of extra parameters in any one of these models should also be considered.  For example I can estimate risk ratios accurately from an odds ratio-based logistic regression model if I include approximately correct interactions in the logistic model.  Examples of this may be found at fharrell.com.

