<p>This is more a related thought than anything else, but I thought it might be useful for people looking at this question in the future. Herr K. should write a short answer and have it marked as accepted.</p>

<p>Let $X$ be an $n \times k$ matrix with linearly independent columns. Let $S \equiv \text{span}(X)$.</p>

<p>Regression can be thought of as the following problem: Given an n-dimensional vector $y \in \mathcal{R}^n$, find the vector in $\hat{y} \in S$ that is closest to $y$ -- i.e.
$$\hat{y} = \arg\min_{z \in \mathcal{R}^n} ||y - z||$$
It can be shown that the solution to this problem is given by
$$\hat{y} = Py = X (X' X)^{-1} X' y$$
where we refer to the matrix $P = X (X' X)^{-1} X'$ as the "projector matrix." Then the matrix $M = (I - P)$ allows you to get the residuals of your regression by $\hat{u} = (I - P)y = y - \hat{y}$. As you said, if $X$ is square then $P$ reduces to the identity and the residuals are 0. This is insightful for several reasons.</p>

<ul>
<li>Geometrically, if $X$ is square and has $n$ linearly independent columns then $\text{span}(X)$ is all of $\mathcal{R}^n$. Then the projection problem reduces to picking the vector in $\hat{y} \in \mathcal{R}^n$ that is closest to some vector in $y \in \mathcal{R}^n$. Obviously, if your vector is coming from the span of the set, the minimum distance choice would be the vector itself. We see this because as you showed $P = I$ in this case. Then $M = (I - P) = 0$ which means you have 0 residuals and you have a perfect fit of your data.</li>
<li>Additionally, this is why empirical economists are careful about "throwing in the kitchen sink" in terms of how many variables are included. If you have as many linearly independent variables as observations then you can get a perfect match whether your exogenous variables are related to your endogenous variables or not.</li>
</ul>

CC BY-SA 3.0
2016-03-25T19:37:28.533Z
2016-03-25T19:37:28.533Z
cc7768
<p>The inverse of some matrix $X$, $X^{-1}$, is defined for square matrices only (i.e. when $X$ has the same number of rows and columns). In the typical econometric applications, the data matrix $X$ usually has far more rows (observations) than columns (regressors). Formally speaking, the matrix $X$ in your definition of $M$ has dimension $n\times k$ but $n\ne k$ (actually $n\gg k$), and so it's not a square matrix. </p>

<p>Therefore the second equality in your derivation is not valid; in particular, $(X'X)^{-1}\ne X^{-1}(X')^{-1}$, because $X^{-1}$ and $(X')^{-1}$ are not defined.</p>

CC BY-SA 3.0
2016-04-25T03:58:25.590Z
2016-04-25T03:58:25.590Z
Herr K.
<p>I am struggling to understand why M is not null since:
$$\mathbf M=I−X(X′X){^-}^1X′=I−XX{^-}^1X'{^-}^1X′=I-I=[0] $$
What's wrong with that reasonning?</p>

CC BY-SA 3.0
2016-03-25T18:53:14.490Z
Because $X$ need not be square matrix, and so $(X'X)^{-1}\ne X^{-1}X'^{-1}$. You can't invert a non-square matrix.
Herr K.
CC BY-SA 3.0
2016-03-25T18:58:53.693Z
@Herr K.  Thanks! Of course the number of parameters  is not always the number of observations! I do  not know how to mark your comment as the answer though...
jeake
CC BY-SA 3.0
2016-03-26T08:41:34.790Z
You can't. People don't get points for good comments. They are just there to be helpful in simple cases that can be addressed quickly. Just give him a nice smile like this :) It's etiquette ;^)))
Kitsune Cavalry
CC BY-SA 3.0
2016-04-24T20:43:29.807Z
@HerrK It is short, but you could write it up as an answer. (You have my vote...)
Giskard
<econometrics><regression>
Why isn't the "annihilator" matrix a zero matrix?
jeake
# Why isn't the "annihilator" matrix a zero matrix?

(Asked by: username_0 on 2016-03-25)

I am struggling to understand why M is not null since:
$$\mathbf M=I−X(X′X){^-}^1X′=I−XX{^-}^1X'{^-}^1X′=I-I=[0] $$
What's wrong with that reasonning?

**Tags:** econometrics, regression

> **Comments:**
> **username_1**: Because $X$ need not be square matrix, and so $(X'X)^{-1}\ne X^{-1}X'^{-1}$. You can't invert a non-square matrix.  
> **username_0**: @username_1.  Thanks! Of course the number of parameters  is not always the number of observations! I do  not know how to mark your comment as the answer though...  
> **username_2**: You can't. People don't get points for good comments. They are just there to be helpful in simple cases that can be addressed quickly. Just give him a nice smile like this :) It's etiquette ;^)))  
> **username_3**: @HerrK It is short, but you could write it up as an answer. (You have my vote...)  

---

## 2 answers

### Answer by username_1 on 2016-04-25. Score: 3 | Accepted answer

The inverse of some matrix $X$, $X^{-1}$, is defined for square matrices only (i.e. when $X$ has the same number of rows and columns). In the typical econometric applications, the data matrix $X$ usually has far more rows (observations) than columns (regressors). Formally speaking, the matrix $X$ in your definition of $M$ has dimension $n\times k$ but $n\ne k$ (actually $n\gg k$), and so it's not a square matrix.

Therefore the second equality in your derivation is not valid; in particular, $(X'X)^{-1}\ne X^{-1}(X')^{-1}$, because $X^{-1}$ and $(X')^{-1}$ are not defined.


---

### Answer by username_4 on 2016-03-25. Score: 4

This is more a related thought than anything else, but I thought it might be useful for people looking at this question in the future. username_1. should write a short answer and have it marked as accepted.

Let $X$ be an $n \times k$ matrix with linearly independent columns. Let $S \equiv \text{span}(X)$.

Regression can be thought of as the following problem: Given an n-dimensional vector $y \in \mathcal{R}^n$, find the vector in $\hat{y} \in S$ that is closest to $y$ -- i.e.
$$\hat{y} = \arg\min\_{z \in \mathcal{R}^n} ||y - z||$$
It can be shown that the solution to this problem is given by
$$\hat{y} = Py = X (X' X)^{-1} X' y$$
where we refer to the matrix $P = X (X' X)^{-1} X'$ as the "projector matrix." Then the matrix $M = (I - P)$ allows you to get the residuals of your regression by $\hat{u} = (I - P)y = y - \hat{y}$. As you said, if $X$ is square then $P$ reduces to the identity and the residuals are 0. This is insightful for several reasons.

* Geometrically, if $X$ is square and has $n$ linearly independent columns then $\text{span}(X)$ is all of $\mathcal{R}^n$. Then the projection problem reduces to picking the vector in $\hat{y} \in \mathcal{R}^n$ that is closest to some vector in $y \in \mathcal{R}^n$. Obviously, if your vector is coming from the span of the set, the minimum distance choice would be the vector itself. We see this because as you showed $P = I$ in this case. Then $M = (I - P) = 0$ which means you have 0 residuals and you have a perfect fit of your data.
* Additionally, this is why empirical economists are careful about "throwing in the kitchen sink" in terms of how many variables are included. If you have as many linearly independent variables as observations then you can get a perfect match whether your exogenous variables are related to your endogenous variables or not.


---

