Variable with negative coefficient in Ridge Regression and positive correlation In my research, I have four independent variables (X1, X2, X3 and X4) and one response variable (Y). Upon checking the VIF values of the explanatory variables, I noticed that they have high multicollinearity, so I decided to use a Ridge Regression to make predictions about the variables, without, of course, using their P-values. When I plotted the Ridge Trace Plot and the chart with the regression coefficients from the Ridge Regression outcome, I noticed that X2 and X4 had negative effects on the model.
I wasn't surprised that this would happen with X4, but I didn't expect the same to happen with X2. I was even more puzzled when I performed a simple linear regression comparing Y and X2, and noticed a positive correlation.
So I decided to remove X3 from my study, since this variable had the highest VIF value (about 13), and I realized that everything made more sense, with X2 presenting a positive influence on the model.
Should I then delete X3 permanently? If I can include this variable in the model, how would I explain the fact that X2 has a negative effect on the model, and yet it has a positive correlation with Y?

Here's an image that explains psboonstra's idea of the situation being a case of the Yule-Simpson effect.
I simulated $X_1 \sim N(0,1)$ and $Y = X_1 + \epsilon$ with $\epsilon \sim N(0,1)$, and added an extra variable $X_2 = 5+\text{round}((Y+X_1)/2)$
set.seed(1)
n = 300
X1 = rnorm(n,0,1)
noise = rnorm(n,0,1)
Y = X1+noise
X2 = 5+round((X1+Y)/2)
plot(X1,Y, col = X2, pch = 20)
lm(Y~X1+X2)


In this case the variable $X_2$ is discrete such that one can see it still as a categorical variable as in common examples of the effect.
The Yule-Simpson effect is that the positive effect of $X_1$ on $Y$ reverses to a negative effect when you consider the "groups" $X_2$ as well.
With some imagination one can see the variable $X_2$ becoming continuous and giving a continuous range of groups.
