<p>From a) we obtain <span class="math-container">$y - \hat{y}_x = M_xy = M_xu$</span> and from b) <span class="math-container">$y - \hat{y}_z = M_zy = M_zX\beta + M_zu$</span>, where <span class="math-container">$M_x$</span> and <span class="math-container">$M_z$</span> are the residual-maker matrices (the ones from the accepted answer of the question that you linked) and <span class="math-container">$\hat{y}_x$</span> and <span class="math-container">$\hat{y}_z$</span> are the predicted values of y based on the who linear models.</p>

<p>The estimate of the variance is  <span class="math-container">$s_x^2 = \frac{(y - \hat{y}_x)^T(y - \hat{y}_x)}{n-k}$</span> for the first model and <span class="math-container">$s_z^2 = \frac{(y - \hat{y}_z)^T(y - \hat{y}_z)}{n-r}$</span> for the second.</p>

<p>Thus <span class="math-container">$E(s_z^2) = \frac{E[(M_zX\beta + M_zu)^T(M_zX\beta + M_zu)]}{n-r} = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zX\beta)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[(M_zX\beta)^T(M_zX\beta)] + 2E[(M_zX\beta)^T(M_zu)] + E[(M_zu)^T(M_zu)]) = \frac{1}{n-r}( E[||M_zX\beta||_2^2] + 2E[\beta^TX^TM_zu] + E[(M_zu)^T(M_zu)]),$</span></p>

<p>as <span class="math-container">$(M_zX\beta)^T(M_zu) = (M_zu)^T(M_zX\beta) =  \beta^TX^TM_z^TM_zu = \beta^TX^TM_zu \hspace{.2cm} \in \mathbb{R}.$</span></p>

<p>Beacuse <span class="math-container">$Z, X -$</span> fixed <span class="math-container">$||M_zX\beta||_2^2$</span> and <span class="math-container">$\beta^TX^TM_z$</span> are constants.</p>

<p>Thus <span class="math-container">$$E(s_z^2) = \frac{1}{n-r}(||M_zX\beta||_2^2 + 2\beta^TX^TM_zE(u) + E(u^TM_zu)) =$$</span> <span class="math-container">$$= \frac{||M_zX\beta||_2^2}{n-r} + 0 + \frac{E(u^TM_zu)}{n-r} = \frac{||M_zX\beta||_2^2}{n-r} + \sigma^2 \neq \sigma^2 \hspace{.2cm} in \hspace{.1cm}general.$$</span></p>

CC BY-SA 4.0
2019-08-15T21:27:28.533Z
2019-08-15T21:33:24.640Z
2019-08-15T21:33:24.640Z
Grada Gukovic
<p>If the model <span class="math-container">$ (a)  y = X\beta + u $</span> is true, then <span class="math-container">$\omega = u + X\beta - Z\gamma $</span> in <span class="math-container">$(b)$</span>, and the claim follows.</p>

CC BY-SA 4.0
2019-08-14T20:18:33.420Z
2019-08-14T20:18:33.420Z
Bertrand
<p>Consider the two models <span class="math-container">$ (a)  y = X\beta + u $</span> where <span class="math-container">$X$</span> is <span class="math-container">$n \times K$</span> and (b) <span class="math-container">$y = Z\gamma + \omega $</span> where <span class="math-container">$Z$</span> is <span class="math-container">$n \times r$</span>.
Under classical assumptions (and <span class="math-container">$Z$</span> and <span class="math-container">$X$</span> are non-stochastic) if model (a), that is <span class="math-container">$y = X\beta + u$</span> is the true model, show that <span class="math-container">$E(\sigma^{2}_{\omega}) \geq \sigma^2_u$</span> and explain the implication of your result.</p>

<p>I can do it in two cases when <span class="math-container">$r&gt;k$</span> (overfitting) or when <span class="math-container">$r&lt;k$</span> (underfitting). </p>

<p>Is there any short way of doing this, without cases (because for both overfitting and underfitting variance is biased)? So, what I am asking is that can they be clubbed so that result doesn't depend on relationship between <span class="math-container">$r$</span> and <span class="math-container">$k$</span>? </p>

CC BY-SA 4.0
2019-08-14T10:53:41.170Z
If you are asking whether the variance of $w$ is expected to be greater than the variance of $u$, shouldn't you write $\mathrm E [\sigma^2_w] > \mathrm E[\sigma^2_u]$?
ahorn
CC BY-SA 4.0
2019-08-14T11:52:40.707Z
No, because a) is a true model (resembling population) and b) is misspecified model, direction of misspecification is not given. That's why I took cases. But since in both the cases expected variance will be more than true variance (biased estimator). So, my ques is can these be clubbed ie. Can we show this without considering whether r is bigger or smaller to K. If so, how can we do it?
Elina Gilbert
CC BY-SA 4.0
2019-08-14T12:46:45.390Z
E. Sommer understood what I was asking. They made the appropriate edit to the notation for that inequality.
ahorn
CC BY-SA 4.0
2019-08-14T16:15:51.283Z
What do you mean by the "variance is biased"? If you estimate $\sigma^2$ by $s^2$ then the last random variable is unbiased for $\sigma^2$.
Bertrand
<regression>
Specification bias - estimated variance is biased estimator of true variance of error term
Elina Gilbert
# Specification bias - estimated variance is biased estimator of true variance of error term

(Asked by: username_0 on 2019-08-14)

Consider the two models $ (a) y = X\beta + u $ where $X$ is $n \times K$ and (b) $y = Z\gamma + \omega $ where $Z$ is $n \times r$.
Under classical assumptions (and $Z$ and $X$ are non-stochastic) if model (a), that is $y = X\beta + u$ is the true model, show that $E(\sigma^{2}\_{\omega}) \geq \sigma^2\_u$ and explain the implication of your result.

I can do it in two cases when $r>k$ (overfitting) or when $r<k$ (underfitting).

Is there any short way of doing this, without cases (because for both overfitting and underfitting variance is biased)? So, what I am asking is that can they be clubbed so that result doesn't depend on relationship between $r$ and $k$?

**Tags:** regression

> **Comments:**
> **username_1**: If you are asking whether the variance of $w$ is expected to be greater than the variance of $u$, shouldn't you write $\mathrm E [\sigma^2_w] > \mathrm E[\sigma^2_u]$?  
> **username_0**: No, because a) is a true model (resembling population) and b) is misspecified model, direction of misspecification is not given. That's why I took cases. But since in both the cases expected variance will be more than true variance (biased estimator). So, my ques is can these be clubbed ie. Can we show this without considering whether r is bigger or smaller to K. If so, how can we do it?  
> **username_1**: E. Sommer understood what I was asking. They made the appropriate edit to the notation for that inequality.  
> **username_2**: What do you mean by the "variance is biased"? If you estimate $\sigma^2$ by $s^2$ then the last random variable is unbiased for $\sigma^2$.  

---

## 2 answers

### Answer by username_3 on 2019-08-15. Score: 1 | Accepted answer

From a) we obtain $y - \hat{y}\_x = M\_xy = M\_xu$ and from b) $y - \hat{y}\_z = M\_zy = M\_zX\beta + M\_zu$, where $M\_x$ and $M\_z$ are the residual-maker matrices (the ones from the accepted answer of the question that you linked) and $\hat{y}\_x$ and $\hat{y}\_z$ are the predicted values of y based on the who linear models.

The estimate of the variance is $s\_x^2 = \frac{(y - \hat{y}\_x)^T(y - \hat{y}\_x)}{n-k}$ for the first model and $s\_z^2 = \frac{(y - \hat{y}\_z)^T(y - \hat{y}\_z)}{n-r}$ for the second.

Thus $E(s\_z^2) = \frac{E[(M\_zX\beta + M\_zu)^T(M\_zX\beta + M\_zu)]}{n-r} = \frac{1}{n-r}( E[(M\_zX\beta)^T(M\_zX\beta)] + E[(M\_zX\beta)^T(M\_zu)] + E[(M\_zu)^T(M\_zX\beta)] + E[(M\_zu)^T(M\_zu)]) = \frac{1}{n-r}( E[(M\_zX\beta)^T(M\_zX\beta)] + 2E[(M\_zX\beta)^T(M\_zu)] + E[(M\_zu)^T(M\_zu)]) = \frac{1}{n-r}( E[||M\_zX\beta||\_2^2] + 2E[\beta^TX^TM\_zu] + E[(M\_zu)^T(M\_zu)]),$

as $(M\_zX\beta)^T(M\_zu) = (M\_zu)^T(M\_zX\beta) = \beta^TX^TM\_z^TM\_zu = \beta^TX^TM\_zu \hspace{.2cm} \in \mathbb{R}.$

Beacuse $Z, X -$ fixed $||M\_zX\beta||\_2^2$ and $\beta^TX^TM\_z$ are constants.

Thus $$E(s\_z^2) = \frac{1}{n-r}(||M\_zX\beta||\_2^2 + 2\beta^TX^TM\_zE(u) + E(u^TM\_zu)) =$$ $$= \frac{||M\_zX\beta||\_2^2}{n-r} + 0 + \frac{E(u^TM\_zu)}{n-r} = \frac{||M\_zX\beta||\_2^2}{n-r} + \sigma^2 \neq \sigma^2 \hspace{.2cm} in \hspace{.1cm}general.$$


---

### Answer by username_2 on 2019-08-14. Score: 1

If the model $ (a) y = X\beta + u $ is true, then $\omega = u + X\beta - Z\gamma $ in $(b)$, and the claim follows.


---

