Find the volume of the solid bounded by $z^2=xy$; $x+y=a$; $x+y=b$ $(0Find the volume of the solid bounded by $z^2=xy$; $x+y=a$; $x+y=b$ $(0<a<b)$ by applying variable substitution. $xy>0$ and $x,y$ can't be negative at the same time so $x>0;y>0.$ Now, $a - x < y < b - x$ and we get that $a < x < b$. So the integral to compute is $\int_{a}^{b}\int_{a-x}^{b-x}{\sqrt{xy}\,dxdy}$. Now this integral, as requested by the problem, should be solved by applying variable substitution. But the answer in the textbook is $\dfrac{\pi(b^3-a^3)}{12}$. I don't get where this $\pi$ comes from. The only way I know to obtain such a result is by changing to polar coordinate system, but I don't see any way to obtain relations between $a,b$, $a-x$, $b-x$ and the new coordinate system especially because in none of these equations are $x$ or $y$ squared. Am I doing some mistake or how does this $\pi$ appear in the answer? Thanks in advance.

The quadric $z^2 = x y $ corresponds to the quadratic form
$ r^T Q r = 0 $
where $r = [x, y, z]^T$ and
$Q = \begin{bmatrix} 0 && -1/2 && 0 \\ -1/2 && 0 && 0 \\ 0 && 0 && 1 \end{bmatrix} $
Diagonalizing $Q$ it becomes $Q = R D R^T $ where
$ D = \begin{bmatrix} -0.5 && 0 && 0 \\ 0 && 0.5 && 0 \\ 0 && 0 && 1 \end{bmatrix} $
$ R = \begin{bmatrix} \dfrac{1}{\sqrt{2}} && - \dfrac{1}{\sqrt{2}} && 0 \\ \dfrac{1}{\sqrt{2}} && \dfrac{1}{\sqrt{2}} && 0 \\ 0 && 0 && 1 \end{bmatrix} $
Define the change of variable as follows:
$ w = R^T r $ i.e. $r = R w $.  So $x + y = [1, 1, 0] r = [1, 1, 0] R w = [\sqrt{2}, 0, 0 ] w $
The equation of the quadric is
$ 0.5 w_1^2 = 0.5 w_2^2 + w_3^2 $
which simplifies to
$ w_1^2 = w_2^2 + 2 w_3^2 $
Now the equation of the boundary $r^T Q r = 0 $ is a cone whose axis is along the the first column of $R$, so in the changed variable it is a cone whose axis is the $w_1$ axis.  The limits of $w_1$ are from $\dfrac{a}{\sqrt{2}} $ to $\dfrac{b}{\sqrt{2}} $.  If we take the volume between $w_1 = 0$ and $w_1=c$, then the volume is just one third of the area of base (in the $w_2, w_3$ plane) times the height $c$; the area is $ \pi \left( \dfrac{c^2}{\sqrt{2}} \right) $ so that volume is
$ \dfrac{1}{3 \sqrt{2}} \pi c ^ 3 $
Now we want to take the difference of volumes between $c = \dfrac{b}{\sqrt{2}}$ and $c = \dfrac{a}{\sqrt{2}} $ , and this is
$ V = \dfrac{\pi}{3\sqrt{2}} \left( \dfrac{b^3 }{2 \sqrt{2}} - \dfrac{a^2}{2 \sqrt{2}} \right) = \dfrac{ \pi (b^3 - a^3) }{12}$
