Can we remove significant variables in a regression? I first ran a regression with all the variables and then ran the regression again with only significant variables (or the variables of interest).
One of the variables in the 2nd regression is highly correlated with another variable of interest, though both of them are significant but are highly correlated. And this is causing the sign of the coefficient of other variables to go opposite way (not as per expectation or theory).
My question is that can we still remove one of those (correlated) significant variables? It makes the whole result come as per the expectations.
Thanks


And this is causing the sign of the coefficient of other variables to go opposite way (not as per expectation or theory).

Basically (i.e. this is an oversimplification), this means that the effect of the second variable is negative, once the effect of the first is controlled for. For example, suppose you're doing a regression on life span, and one of your variables is diabetes diagnosis, and another is use of insulin. You'll probably find that insulin use is negatively correlated with life span. But when you include diabetes diagnosis, the effect will probably become positive. That's because someone who has diabetes and is using insulin has a lower life expectancy when compared to the general population, but a higher life expectancy when compared to someone with diabetes but isn't taking their insulin shots.

My question is that can we still remove one of those (correlated) significant variables? It makes the whole result come as per the expectations.

Well, what the goal of the regression? To gain insight into reality, or to come up with numbers that accord with your expectations? If having a negative coefficient really bothers you, you can look at what happens when you do PCA and do regression on the principal components rather than the original variables. In my example above, "has diabetes and doesn't take insulin" would probably be the first component, and "takes the medically appropriate level of insulin" the second. Looking at what the components come out to be and what their coefficients may give you more insight into what's going on, and result in the coefficients "making sense" more.
The lesson here is that properly interpreting regression is complicated, and involves more than just looking at the coefficients. Just saying "the coefficient for a variable represents how much effect that variable has on the response variable" is a simplification that sometimes is very misleading.
