Understanding multiple linear regression residuals I'm a bit confused whether residual vectors in OLS are orthogonal to every vector in X.
The problem I have is: Which of the following are true statements for our multiple linear regression with datapoints as rows of design matrix X, label vector Y, and using the model Ŷ = X̂.
I can understand that the residual vector e = Y - Ŷ is orthogonal to Y. And to me, it would make sense for the residual vector e = Y - Ŷ to also be orthogonal to some cases of vector in X. But what about every?
Any help would be appreciated.
I reference this diagram:


Its been a long time since I've had to do some linear algebra, so forgive me if I've forgotten some of it.  Let's begin by assuming that $X$ is full rank and $\operatorname{rnk}(X)=p<n$.
Minimizing the loss means
$$ \nabla R = 0 \>.$$
Note that $\nabla R$ in terms of $X$ and $y$ is
$$ \nabla R = -2X^T (y- X\theta) $$
Note that $e = y - X\theta$.  This means that $y-X\theta$ is in the null space of $X^T$, so the residual is orthogonal to the columns of $X$.  Those columns span some subspace of $\mathbb{R}^n$ (the blue plane in your image).  So since the residual is orthogonal to the columns, and the columns are linearly independent, it is orthogonal to all linear combinations of those columns. In particular, it is orthogonal to the prediction (the closest vector in the plane to $y$).
