Q:

Understanding linear regression

In class we've seen that $a$ (the weights) must satisfy $$X^T (y-Xa) =0$$
Here $X$ is a $(n\times d)$ matrix (so we have $n$ samples in $\mathbb R^d$)
let's denote the residuals $r = y-Xa$. In our class notes, it is written that

The normal equations require the residuals to be orthogonal to each of the columns of $X$.

Why?

Therefore, the solution of the linear regression is a projection of $y$ onto the subspace spanned by $v_1 , \ldots , v_d$ (the columns of $X$)

Can you please explain this?

A:

Recall that the squared error loss you are trying to minimize is
$$
(y-Xa)^T (y-Xa) = y^Ty - 2a^TX^Ty + a^TX^TXa;
$$
if you take all the derivatives with respect to each element of $a$ and arrange them into a column vector, you get
$$
-2X^Ty + 2X^TXa.
$$
Then you set that equal to zero to get a necessary condition for the minimization of the sum of squares. These are the "normal equations," and it is the expression you wrote earlier:
$$
X^T (y-Xa) =0.
$$
When you solve for $a$ you get $\hat{a} = (X^TX)^{-1}X^Ty$, which makes the fitted values (your quote refers to these as "solutions")
$$
\hat{y} = X\hat{a} = X(X^TX)^{-1}X^Ty.
$$
The matrix $d \times d$ matrix $X(X^TX)^{-1}X^T$ is known as the projection matrix. For more information on why it is called that, see here.

