Does the fraction of explained variance in the first PCA component necessarily drop if you remove features from the dataset? When performing PCA on a certain dataset and logging the fraction of total variance explained by the components, will this fraction drop for the first component if one or more features (that were present in the component's loadings) are removed and the PCA re-computed?
Asked by one of my students. I said yes but now I have a doubt ...

Often that will happen, but not necessarily.  When the feature that is removed constitutes most of the first PC, you then are basically doing PCA on everything else.  The new first PC will be close to the second original PC and its fraction of the total variance could be just about anything $1/(d-1)$ or larger when there are $d-1$ variables left.  For $d\ge 3$ this raises the possibility of a decrease in the variance proportion.
Let us, then, produce the smallest possible example, and let's make it simple.  I begin with a large vector $(10,0,0)^\prime.$  Now adjoin two simple smaller vectors, say
$$X = \pmatrix{10&0&0\\0&1&1\\0&1&-1}.$$
Doing PCA directly on this matrix (no centering, no scaling) shows the first PC accounts for $100/(100+2+2) \approx 96.15\%$ of the total variance.  Removing the first column gives two equal-size orthogonal columns with two PCs each (therefore) contributing $50\%$ to the total.
