Lasso Selecting Different Features Depending on Weighting I am performing a logistic regression with an L1/Lasso penalty.
To exemplify: I am attempting to predict a variable $Y$ as a function of $\vec{X} = [X_1, X_2, X_3]$. Depending on the weight of my penalty Lasso obviously selects a different number of features from $\vec{X}$ to use. What confuses me is that for one penalty weighting the regression might select $X_1$ and for a lower weighting it might select $X_2$ and $X_3$but not $X_1$.
How does this happen that one variable has the most explanatory power when only one variable is selected but is "less useful" when we can select more variables? How does this make sense or shouldn't it?

This can happen when you have multi-colinearity between your variables.
Imagine $X_1=X_2+X_3$ and $Y=2X_1 + 0.2X_2 + 0.1X_3$. So most of the predictive power is contained in $X_1$ and we can do a pretty good one variable prediction with just $X_1$. However, if we're allowed to have two variables we can do a better job with $X_2$ and $X_3$ since there's further information here that's not contained within $X_1$.
