<p>Comparing simple or multiple regressions estimates is in itself interesting. There are only two special cases where simple regression of <span class="math-container">$y$</span> on <span class="math-container">$x_1$</span> will produce the same OLS estimate on <span class="math-container">$x_1$</span> as regressing <span class="math-container">$y$</span> on <span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span>. Let's see why.</p>
<p><span class="math-container">$\tilde{y}=\tilde{\beta_o}+\tilde{\beta_1}x_1$</span> and the multiple regression analog <span class="math-container">$\hat{y}=\hat{\beta_o}+\hat{\beta_1}x_1+\hat{\beta_2}x_2$</span>.
There is the following relationship between <span class="math-container">$\tilde {\beta_1}$</span> and <span class="math-container">$\hat{\beta_1}$</span>:
<span class="math-container">$$\tilde{\beta_1}=\hat{\beta_1}+\hat{\beta_2}\tilde{\phi_1}$$</span> where <span class="math-container">$\tilde{\phi_1}$</span> is the slope coefficient of the simple regression of <span class="math-container">$x_{i2}$</span> on <span class="math-container">$x_{i1}$</span>, <span class="math-container">$i=1,...n$</span>.
Therefore, <span class="math-container">$\tilde{\beta_1}$</span> differs from the partial effect of <span class="math-container">$x_1$</span> on <span class="math-container">$\hat{y}$</span>. The confounding term is the partial effect of <span class="math-container">$x_2$</span> on <span class="math-container">$\hat{y}$</span> times the slope in the sample regression of <span class="math-container">$x_2$</span> on <span class="math-container">$x_1$</span>.</p>
<p>There are two distinct cases where they are equal:</p>
<ul>
<li>the partial effect of <span class="math-container">$x_2$</span> on <span class="math-container">$\hat{y}$</span> is zero in the sample (<span class="math-container">$\hat{\beta_2}=0$</span>)</li>
<li><span class="math-container">$x_1$</span> and <span class="math-container">$x_2$</span> are uncorrelated in the sample (<span class="math-container">$\tilde{\phi_1}=0$</span>)</li>
</ul>
<p>Showing this omitted variable bias in general requires a bit of matrix algebra and is not important here. All I want to show is that if you assume that both play a role, leaving either out, will lead in biased estimates. That is why defining an appropriate model is actually quite difficult. Not because of causality (alone) but really because correlation alone is not what matters in regression analysis. The simple regression result of <span class="math-container">$$\frac{sample  \ covariance \ of \ x \ and \ y}{sample \ variance \ of \ x}$$</span> only works if the two conditions above are fulfilled. Otherwise, your estimator is biased.</p>
<p>What you want to address is <a href="https://en.wikipedia.org/wiki/Multicollinearity" rel="nofollow noreferrer">multicollinearity</a>. The wikipedia article also offers guidance for <a href="https://en.wikipedia.org/wiki/Multicollinearity#Detection" rel="nofollow noreferrer">detection</a> as well as it's implication.</p>
<p>I have never modelled anything related to laws but posing a question and answering it with statistics is a delicate and complex task. I usually follow something along the line of:</p>
<ul>
<li><p>What do I try to achieve?</p>
</li>
<li><p>What is my hypothesis? What is the (economic) theory behind it?</p>
</li>
<li><p>Has this been asked before somewhere? If so, what did they use and
why?</p>
</li>
<li><p>What kind of model will I need to use? GLM (OLS), ML, ARIMA,... and what functional form is best suited for this.</p>
</li>
<li><p>What data will be needed to answer this (and satisfy the
assumptions of the model of choice)</p>
</li>
<li><p>How do I need to clean, transform and check my data before I can use it. Is it stationary? Is it noisy? Any structural breaks (Paul Volker, Great Moderation, Dot.com bubble, Subprime crisis, Covid crises, in your case new laws, to name a few)? How do I account for these regime switches?</p>
</li>
<li><p>Is there a different factor influencing this.</p>
</li>
<li><p>Am I at risk of omitted variable bias? Or multicollinearity?</p>
</li>
<li><p>What tests are best suited to check for the problems above? How to check for stationarity, collinearity, ...</p>
</li>
<li><p>Once a model is setup, data is collected and appropriately transformed, you check your results. For example, is my error term uncorrelated with me explanatory variable(s). If not, what did I miss. Redo the above.</p>
</li>
<li><p>Am I overdoing it now (data mining)?</p>
</li>
<li><p>How do the results compare to existing findings? How do I interpret them?</p>
</li>
</ul>
<p>A correlation of 0.9728 is not something I think should be used.</p>

CC BY-SA 4.0
2021-09-07T21:58:38.717Z
"Not because of causality (alone) but really because correlation alone is not what matters in regression analysis". Do you mean not causality or correlation is what matter, but we need to concern and weight both ?
Phil Nguyen
CC BY-SA 4.0
2021-09-07T22:14:02.597Z
Yes, both, but also if the relationship is actually linear. Look up "anscombe quartet" to see how misleading correlation can be.
AKdemy
CC BY-SA 4.0
2021-09-07T20:34:33.190Z
2021-09-07T20:34:33.190Z
AKdemy
<p>In my case, I have two questions that I want to ask here relating to correlations among variables:</p>
<ol>
<li><p>Whether it makes sense if we test the impact of laws on two outcome variables that are correlated to each other (around 0.6)</p>
</li>
<li><p>I have a table of correlation of some independent variables as below
<a href="https://i.sstatic.net/PP4Yt.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/PP4Yt.png" alt="enter image description here" /></a></p>
</li>
</ol>
<p>Note: The variables names on the first row are equal to those on the column, for example (workpl~s=workplaces, sorry for the inconvenience due to the character limitation in STATA)</p>
<p>I am wondering which variables should be excluded from the regression? Is it okay to retain all of them in the regression. From my understanding, between two variables having high correlation, we should exclude one. High correlation is defined by having a correlation value higher than 0.5 and lower than 1.</p>

<econometrics>
When should we exclude an independent variable out of the regression?
Phil Nguyen
# When should we exclude an independent variable out of the regression?

(Asked by: username_0 on 2021-09-07)

In my case, I have two questions that I want to ask here relating to correlations among variables:

1. Whether it makes sense if we test the impact of laws on two outcome variables that are correlated to each other (around 0.6)
2. I have a table of correlation of some independent variables as below
   [![enter image description here](https://i.sstatic.net/PP4Yt.png)](https://i.sstatic.net/PP4Yt.png)

Note: The variables names on the first row are equal to those on the column, for example (workpl~s=workplaces, sorry for the inconvenience due to the character limitation in STATA)

I am wondering which variables should be excluded from the regression? Is it okay to retain all of them in the regression. From my understanding, between two variables having high correlation, we should exclude one. High correlation is defined by having a correlation value higher than 0.5 and lower than 1.

**Tags:** econometrics


---

## 1 answers

### Answer by username_1 on 2021-09-07. Score: 3 | Accepted answer

Comparing simple or multiple regressions estimates is in itself interesting. There are only two special cases where simple regression of $y$ on $x\_1$ will produce the same OLS estimate on $x\_1$ as regressing $y$ on $x\_1$ and $x\_2$. Let's see why.

$\tilde{y}=\tilde{\beta\_o}+\tilde{\beta\_1}x\_1$ and the multiple regression analog $\hat{y}=\hat{\beta\_o}+\hat{\beta\_1}x\_1+\hat{\beta\_2}x\_2$.
There is the following relationship between $\tilde {\beta\_1}$ and $\hat{\beta\_1}$:
$$\tilde{\beta\_1}=\hat{\beta\_1}+\hat{\beta\_2}\tilde{\phi\_1}$$ where $\tilde{\phi\_1}$ is the slope coefficient of the simple regression of $x\_{i2}$ on $x\_{i1}$, $i=1,...n$.
Therefore, $\tilde{\beta\_1}$ differs from the partial effect of $x\_1$ on $\hat{y}$. The confounding term is the partial effect of $x\_2$ on $\hat{y}$ times the slope in the sample regression of $x\_2$ on $x\_1$.

There are two distinct cases where they are equal:

* the partial effect of $x\_2$ on $\hat{y}$ is zero in the sample ($\hat{\beta\_2}=0$)
* $x\_1$ and $x\_2$ are uncorrelated in the sample ($\tilde{\phi\_1}=0$)

Showing this omitted variable bias in general requires a bit of matrix algebra and is not important here. All I want to show is that if you assume that both play a role, leaving either out, will lead in biased estimates. That is why defining an appropriate model is actually quite difficult. Not because of causality (alone) but really because correlation alone is not what matters in regression analysis. The simple regression result of $$\frac{sample \ covariance \ of \ x \ and \ y}{sample \ variance \ of \ x}$$ only works if the two conditions above are fulfilled. Otherwise, your estimator is biased.

What you want to address is [multicollinearity](https://en.wikipedia.org/wiki/Multicollinearity). The wikipedia article also offers guidance for [detection](https://en.wikipedia.org/wiki/Multicollinearity#Detection) as well as it's implication.

I have never modelled anything related to laws but posing a question and answering it with statistics is a delicate and complex task. I usually follow something along the line of:

* What do I try to achieve?
* What is my hypothesis? What is the (economic) theory behind it?
* Has this been asked before somewhere? If so, what did they use and
  why?
* What kind of model will I need to use? GLM (OLS), ML, ARIMA,... and what functional form is best suited for this.
* What data will be needed to answer this (and satisfy the
  assumptions of the model of choice)
* How do I need to clean, transform and check my data before I can use it. Is it stationary? Is it noisy? Any structural breaks (Paul Volker, Great Moderation, Dot.com bubble, Subprime crisis, Covid crises, in your case new laws, to name a few)? How do I account for these regime switches?
* Is there a different factor influencing this.
* Am I at risk of omitted variable bias? Or multicollinearity?
* What tests are best suited to check for the problems above? How to check for stationarity, collinearity, ...
* Once a model is setup, data is collected and appropriately transformed, you check your results. For example, is my error term uncorrelated with me explanatory variable(s). If not, what did I miss. Redo the above.
* Am I overdoing it now (data mining)?
* How do the results compare to existing findings? How do I interpret them?

A correlation of 0.9728 is not something I think should be used.

> **Comments:**
> **username_0**: "Not because of causality (alone) but really because correlation alone is not what matters in regression analysis". Do you mean not causality or correlation is what matter, but we need to concern and weight both ?  
> **username_1**: Yes, both, but also if the relationship is actually linear. Look up "anscombe quartet" to see how misleading correlation can be.  

---

