So I am missing why this proves what it is intended to prove, but I can certainly fill in your missing details.
So the claim is that we examine a perturbation$\delta r$ which is very very small and for any other quantity we define $\delta F = F(r + \delta r)-F(r)$ to first order.
We insert it in every single term in the integral (this is the part which I find suspicious) and have $$\delta E =\int\mathrm dM\left(\delta u - \frac{GM}{r+\delta r} + \frac{GM}{r} \right).$$ To answer your first question, the middle term satisfies $$\begin{align} - \frac{GM}{r+\delta r} &= -\frac{GM}r \frac1{1+\frac{\delta r}r}\ &= -\frac{GM}r \left( 1 - \frac{\delta r}r + \left(\frac{\delta r}r\right)^2 -\dots \right) \end{align}$$ by the usual rules of geometric series. To first order we keep the first two terms only.
Second there is the claim that $$\delta u = - p~\delta V,$$a standard claim of thermodynamics which again just sort of doesn't make sense here but that's because I don't see exactly what the argument is. Since $u$ is the internal energy per unit of mass then $V$ should be the volume per unit of mass, which makes it exactly $V = 1/\rho$ where $\rho$ is the mass density. 649
Doing a <PERSON> series $$\delta\frac1\rho =-\frac{\rho'~\delta\rho}{\rho^2}$$ to first order.
I think I can make sense of this but I do not think I can make sense of this inside of that integral sign which I would want to rewrite as $$\int \mathrm dM =\int \rho(r)\mathrm dr$$ and then I would never do a Lagrangian path variation on $r$ but on $\rho$ maybe. I spent a little time trying to Google ways to do thermodynamics as a sort of calculus of variations problem, and got a lot of black hole thermodynamics papers which figures. I suppose one could try to maximize the entropy or so, maybe minimize a free energy assuming that the core of the star is held at a very constant temperature or so, argue that the density profile which minimizes this should be the quasi-static equilibrium? But again, varying $r$ just sounds like you are doing a fancy version of the normal, “balance two forces against each other” argument and this integral over $\mathrm dM$ is spurious? 474