Should individual $R^2$ of a predictor always be greater than $\Delta R^2$ when removing that predictor from an expanded model? I'm running some regressions with a set of somewhat correlated predictors. Let's call these predictors $x$, $y$ and $z$, and my dependent variable $d$.
I'm focused on the effect of $x$ on $d$.
I first calculate $R^2_x$ by running a regression using only $x$ as a predictor.
I can also calculate $R^2_{full}$ by running a regression using $x$, $y$ and $z$ as predictors.
I calculate $R^2_{reduced}$ by running a regression using only $y$ and $z$ as predictors.
I calculate $\Delta R^2$ as $R^2_{full} - R^2_{reduced}$.
My question is, shouldn't $R^2_x$ always be greater than $\Delta R^2$? I would think that it would, since in the full model, the predictive power of $x$ has been reduced by the inclusion of the correlated predictors $y$ and $z$. However I am observing many instances of $\Delta R^2 > R^2_x$. Is this to be expected? Thanks!

In this answer, I derive a closed-form expression of $R_x^2 - \Delta R^2$ to analyze its sign. Since calculations below heavily rely on geometry/linear algebra, let's first settle down some notations:

*

*$x, y, z, d$ are all viewed as column vectors in $\mathbb{R}^n$.  By convention, we use $e$ to denote the column vector consisting of all ones in $\mathbb{R}^n$ (i.e., the intercept).

*For a vector $x_0 \in \mathbb{R}^n$, we use $\|x_0\|$ to denote its Euclidean norm.

*For any two vectors $x_1, x_2 \in \mathbb{R}^n$, we use $x_1'x_2$ to denote their inner product.

*The subspace of $\mathbb{R}^n$ spanned by vectors $x_1, x_2, \ldots, x_p$ is denoted by $[x_1, x_2, \ldots, x_p]$.  The orthogonal complement space of a subspace $S$ is denoted by $S^\perp$.

*We use $P_M x_0$ to denote the orthogonal projection of $x_0$ onto the subspace $M$.  In particular, if $\{q_1, \ldots, q_m\}$ is an orthogonal basis of $M$, then
\begin{align}
P_Mx_0 = \frac{x_0'q_1}{q_1'q_1}q_1 + \cdots + \frac{x_0'q_m}{q_m'q_m}q_m.
\end{align}
With these notations, various R-squares in the question can be expressed by definition as
\begin{align}
& R_{\text{full}}^2 = \frac{\|P_{[e, x, y, z]}d\|^2}{\|P_{[e]^\perp}d\|^2}, \\
& R_{\text{reduced}}^2 = \frac{\|P_{[e, y, z]}d\|^2}{\|P_{[e]^\perp}d\|^2}, \\
& R_{x}^2 = \frac{\|P_{[e, x]}d\|^2}{\|P_{[e]^\perp}d\|^2}, \\
& \Delta R^2 = R_{\text{full}}^2 - R_{\text{reduced}}^2 = \frac{\|P_{[e, x, y, z]}d\|^2 - \|P_{[e, y, z]}d\|^2}{\|P_{[e]^\perp}d\|^2}.
\end{align}
Therefore, requiring $R_x^2 \geq \Delta R^2$ is equivalent to requiring
\begin{align}
\|P_{[e, x]}d\|^2 \geq \|P_{[e, x, y, z]}d\|^2 - \|P_{[e, y, z]}d\|^2. \tag{1}
\end{align}
By the Gram-Schmidt orthogonalization procedure (see, e.g., Algorithm 3.1 in The Elements of Statistical Learning),  it can be shown that the right hand side of $(1)$ equals to  $\|P_{[r]}d\|^2$, where $r$ is the residual vector after projecting $x$ onto the space $[e, y, z]$.  More specifically, if letting
\begin{align}
& z_1 = e, \\
& z_2 = y - \frac{y'z_1}{n}z_1, \\
& z_3 = z - \frac{z'z_1}{n}z_1 - \frac{z'z_2}{z_2'z_2}z_2,
\end{align}
then (note that $e, z_2, z_3, r$ are mutually orthogonal by construction)
\begin{align}
r = x - \frac{x'z_1}{n}z_1 - \frac{x'z_2}{z_2'z_2}z_2 - \frac{x'z_3}{z_3'z_3}z_3.
\end{align}
Note that $x - \frac{x'z_1}{n}z_1 = P_{[e]^\perp}x$, $\frac{x'z_2}{z_2'z_2}z_2 + \frac{x'z_3}{z_3'z_3}z_3 = P_{[z_2, z_3]}x$, it then follows that
\begin{align}
\|P_{[r]}d\|^2 = \left(d'P_{[e]^\perp}x - d'P_{[z_2, z_3]}x\right)^2.  
\end{align}
Treating the left hand side of $(1)$ similarly, it can be shown that
\begin{align}
\|P_{[e, x]}d\|^2 = (d'e)^2 + (d'P_{[e]^\perp}x)^2.  
\end{align}
Therefore,
\begin{align}
 \|P_{[e, x]}d\|^2 - \|P_{[r]}d\|^2 
= (d'e)^2 + 2d'P_{[e]^\perp}x \times d'P_{[z_2, z_3]}x - (d'P_{[z_2, z_3]}x)^2. \tag{2}
\end{align}
In view of $(2)$, the sign of $\|P_{[e, x]}d\|^2 - \|P_{[r]}d\|^2$ depends on correlations between $x$ and $e, z_2, z_3$, and correlations between $d$ and $e, z_2, z_3$.  Hence it is in general indefinite.
For example, if $x = e + z_2 + z_3 + \epsilon$, where $\epsilon \perp [e, z_2, z_3]$ satisfying $2\|\epsilon\|^2 > \|z_2\|^2$, then
\begin{align}
& P_{[z_2, z_3]}x = z_2 + z_3, \\
& P_{[e]^\perp}x = x - P_{[e]}x = z_2 + z_3 + \epsilon. 
\end{align}
Finally, take $d = z_2 - \epsilon$, then $(2)$ reduces to
\begin{align}
 & \|P_{[e, x]}d\|^2 - \|P_{[r]}d\|^2 \\
=& (d'e)^2 + 2d'(z_2 + z_3 + \epsilon) \times d'(z_2 + z_3) - (d'(z_2 + z_3))^2 \\
=& 2(\|z_2\|^2 - \|\epsilon\|^2) \times \|z_2\|^2 - \|z_2\|^4 \\
=& (\|z_2\|^2 - 2\|\epsilon\|^2)\times \|z_2\|^2 < 0,
\end{align}
hence $R_x^2 < \Delta R^2$.
