Understanding multiple linear regression residuals I'm a bit confused whether residual vectors in OLS are orthogonal to every vector in X.
The problem I have is: Which of the following are true statements for our multiple linear regression with datapoints as rows of design matrix X, label vector Y, and using the model Ŷ = X̂.
I can understand that the residual vector e = Y - Ŷ is orthogonal to Y. And to me, it would make sense for the residual vector e = Y - Ŷ to also be orthogonal to some cases of vector in X. But what about every?
Any help would be appreciated.
I reference this diagram:


Under OLS the residual is orthogonal to every column in the design matrix
To understand this issue, it is worth understanding the concept of the column space of the design matrix $\mathbf{x}$.  This is the space spanned by linear combinations of the column vectors in the design matrix, so it represents every possible vector you can form via the linear transformation $\mathbf{x} \boldsymbol{\theta}$.  In your diagram, the column space is the blue region.
When you fit the regression model using OLS estimation, this method is equivalent to forming the predicted response as the projection of $\mathbf{y}$ onto the column space of $\mathbf{x}$.  In fact, the hat matrix $\mathbf{h} = \mathbf{x} (\mathbf{x}^\text{T} \mathbf{x})^{-1} \mathbf{x}^\text{T}$ is the projection matrix that maps the actual response vector onto the predicted response vector.  This orthogonal projection is shown in the diagram in your question.  Since the fitted model is formed by an orthogonal projection, the residual vector is orthogonal to the column space of $\mathbf{x}$ which means it is orthogonal to every column of $\mathbf{x}$ (and therefore also orthonogal to the predicted response vector $\hat{\mathbf{y}}$).
