  Sparse regression is frequently employed in diverse scientific settings as a
feature selection method. A pervasive aspect of scientific data that hampers
both feature selection and estimation is the presence of strong correlations
between predictive features. These fundamental issues are often not appreciated
by practitioners, and jeapordize conclusions drawn from estimated models. On
the other hand, theoretical results on sparsity-inducing regularized regression
such as the Lasso have largely addressed conditions for selection consistency
via asymptotics, and disregard the problem of model selection, whereby
regularization parameters are chosen. In this numerical study, we address these
issues through exhaustive characterization of the performance of several
regression estimators, coupled with a range of model selection strategies.
These estimators and selection criteria were examined across correlated
regression problems with varying degrees of signal to noise, distribution of
the non-zero model coefficients, and model sparsity. Our results reveal a
fundamental tradeoff between false positive and false negative control in all
regression estimators and model selection criteria examined. Additionally, we
are able to numerically explore a transition point modulated by the
signal-to-noise ratio and spectral properties of the design covariance matrix
at which the selection accuracy of all considered algorithms degrades. Overall,
we find that SCAD coupled with BIC or empirical Bayes model selection performs
the best feature selection across the regression problems considered.

Methodology;Applications