Show that $(ab' - a'b)^2 + 4(ah' - a'h)(bh' - b'h)$ is a perfect square Reference:  A Course of Pure Mathematics (ed. 3) page 38
Show that if $a,a',b,b',h,h'$ are rational and all the values of $x$ and $y$ given by
$ax^2 + 2hxy + by^2 = 1$, $a'x^2 + 2h'xy + b'y^2=1$
are rational, then


*

*$(h - h')^2 - (a-a')(b-b')$ is a perfect square

*$(ab' - a'b)^2 + 4(ah' - a'h)(bh' - b'h)$ is a perfect square
My attempt:
$$ax^2 + 2hxy + by^2 = 1 \\\implies ax^2 + 2hxy + by^2 - (a'x^2 + 2h'xy + b'y^2) = 1 -1 \\\implies (a-a')x^2 + 2(h-h')xy + (b-b')y^2 = 0 \\\implies \frac{1}{y^2}[(a-a')x^2 + 2(h-h')xy + (b-b')y^2] = 0 \\\implies (a-a')\frac{x^2}{y^2} + 2(h-h')\frac{x}{y} + (b-b') = 0$$ Assuming $a-a' \neq 0$, $$\\\implies \frac{x}{y} = \frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'} $$
$\frac{x}{y}$ is always rational if $(h-h')^2 - (a-a')(b-b')$ is a perfect square and vice versa. This proves (1).
$$ax^2 + 2hxy + by^2 = 1$$
$$\implies a (\frac{x}{y})^2 + 2h \frac{x}{y} + b = \frac{1}{y^2}$$
$$\implies a \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}]^2 + 2h \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}] + b = \frac{1}{y^2}$$
Now, if $(h-h')^2 - (a-a')(b-b')$ is a perfect square, then $\frac{x}{y}$ is rational and if $$a \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}]^2 + 2h \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}] + b$$ is a perfect square, then $y$ must be rational. And if $\frac{x}{y}$ and $y$ is rational then $x$ is rational. Therefore it seems to me that if $a-a' \neq 0$ and $(h-h')^2 - (a-a')(b-b')$ is a perfect square and $$a \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}]^2 + 2h \cdot [\frac{-(h-h')\pm \sqrt{(h-h')^2 - (a-a')(b-b')}}{a-a'}] + b$$ is a perfect square, then all values of $x$ and $y$ that satisfy the equations are rational.

Update: Here's a new proof, constructed via resultant theory. I think this is probably what the original poser wanted.
Let $z = x/y$, then
$$(a - a')z^2 + 2(h - h')z + (b - b') = 0$$
$$y^2 = \frac{1}{az^2 + 2hz + b}$$
Define
$$s = 2(a'b - ab')(a - a') + 4(ah' - a'h)(h - h')$$
$$t = (ab' - a'b)^2 + 4(ah' - a'h)(bh' - b'h)$$
Then one can verify that $y$ satisfies a quartic equation
$$ty^4 + sy^2 + (a - a')^2 = 0$$
The solutions to this equation are given by
$$y = \pm\sqrt{-\frac{s}{4t} + \frac{\sqrt{(a - a')^2/t}}{2}} \pm \sqrt{-\frac{s}{4t} - \frac{\sqrt{(a - a')^2/t}}{2}}$$
Since all solutions of $y$ are rational, then
$$\sqrt{\frac{(a - a')^2}{t}}$$
is rational. Hence $\sqrt{t}$ is rational.
Original proof: I assume that in this problem, $x,y$ range over all complex numbers. If complex solutions are ignored, then we can construct counterexamples in which all real solutions are rational, but the second value is negative, hence not a perfect square.
Let $z = x/y$. Suppose that
$$(h - h')^2 - (a - a')(b - b') \neq 0$$
Then there exists two rational roots to the equation
$$(a - a')z^2 + 2(h - h')z + (b - b') = 0$$
Let us call them $z_1,z_2$. Correspondingly, there are four solutions to the original system:
$$\left\{\begin{aligned}x &= \pm\frac{1}{\sqrt{az^2 + 2hz + b}}\\y &= \pm\frac{z}{\sqrt{az^2 + 2hz + b}}\end{aligned}\right.$$
As you have noted, in each case $az^2 + 2hz + b$ must be a perfect square. Suppose that
$$\left\{\begin{aligned}az_1^2 + 2hz_1 + b &= t_1^2\\az_2^2 + 2hz_2 + b &= t_2^2\end{aligned}\right.$$
We find that
$$2h = \frac{(t_2^2 - t_1^2) - a(z_2^2 - z_1^2)}{z_2 - z_1}$$
$$b = t_1^2 - az_1^2 - 2hz_1$$
Now if we write
$$(a - a')z^2 + 2(h - h')z + (b - b') = u(z - z_1)(z - z_2)$$
where $u = a - a'$ is a rational number, then
$$a'z^2 + 2h'z + b' = az^2 + 2hz + b - u(z - z_1)(z - z_2)$$
Matching coefficients of $z$ on both sides, and we obtain expressions for $a',h',b'$.
Plug all the above relations into
$$(ab' - a'b)^2 + 4(ah' - a'h)(bh' - b'h)$$
and simplify the result (this will be very messy, I did it with Mathematica), and it will turn out to be a square of a rational number.
In the above we have assumed that
$$(h - h')^2 - (a - a')(b - b') \neq 0$$
If instead we have
$$(h - h')^2 - (a - a')(b - b') = 0$$
then we can assume
$$\left\{\begin{aligned}h - h' &= k\\a - a' &= p\\b - b' &= q\end{aligned}\right.$$
Then $q = k^2/p$, and
$$\left\{\begin{aligned}h' &= h - k\\a' &= a - p\\b' &= b - k^2/p\end{aligned}\right.$$
Plug these relations into the value, simplify the result, and it will turn out to be a square.
