II. (50 points) Let positive numbers $a, b$ satisfy $a>\frac{b}{\sqrt{2}}$ and such that the inequality about $x$
$$
\sqrt{\dot{x}-1} \geqslant a \sqrt{x+1}-b
$$

always has real solutions. Try to find the range of $f(a, b)=a^{2}-3 a b+b^{2}$.
First, find the conditions that $a$ and $b$ should satisfy. From the original inequality, we get the following equivalent forms: $\sqrt{x-1}+b \geqslant a \sqrt{x+1}$. Squaring both sides and rearranging, we get
$$
\begin{array}{l}
2 b \sqrt{x-1} \geqslant\left(a^{2}-1\right) x+\left(a^{2}-b^{2}+1\right) \text {. } \\
\text { Let } \sqrt{x-1}=t \quad(t \geqslant 0) \text {, then } x=t^{2}+1 \text {, substituting into equation (1) we get } \\
2 b t \geqslant\left(a^{2}-1\right)\left(t^{2}+1\right)+a^{2}-b^{2}+1(t \geqslant 0), \\
\text { i.e., }\left(a^{2}-1\right) t^{2}-2 b t+2 a^{2}-b^{2} \leqslant 0(t \geqslant 0) \text {. } \\
\end{array}
$$

We will discuss 3 cases:
When $a^{2}=1$, equation (2) becomes $-2 b t+2-b^{2} \leqslant 0(t \geqslant 0)$, which has a solution.
When $a^{2}>1$, first we require the discriminant $\Delta \geqslant 0$, i.e.,
$$
(-2 b)^{2}-4\left(a^{2}-1\right)\left(2 a^{2}-b^{2}\right) \geqslant 0 \text {, }
$$

which simplifies to $2 a^{2} \leqslant b^{2}+2\left(a^{2}>1\right)$.
$$
\text { Let } f(t)=\left(a^{2}-1\right) t^{2}-2 b t+2 a^{2}-b^{2} \text {. }
$$

Since $\Delta \geqslant 0$, the equation $f(t)=0$ has two real roots $t_{1}$ and $t_{2}\left(t_{1} \leqslant t_{2}\right)$. Because $t_{1}+t_{2}=\frac{2 b}{a^{2}-1}>0\left(a^{2}>1, b>0\right)$, it follows that $t_{2}>0$. Since the parabola $y=f(t)$ opens upwards, the inequality $f(t) \leqslant 0(t \geqslant 0)$ always has a solution $\max \left\{0, t_{1} \mid \leqslant t \leqslant t_{2}\right.$.
Combining the above, the necessary and sufficient conditions that $a$ and $b$ should satisfy are
$$
2 a^{2} \leqslant b^{2}+2(a>0, b>0),
$$

i.e., $a^{2}-\left(\frac{b}{\sqrt{2}}\right)^{2} \leqslant 1(a>0, b>0)$.
Noting the similarity of equation (5) to the trigonometric identity $\left(\frac{1}{\cos \theta}\right)^{2}-(\tan \theta)^{2}=1$, we let
$$
\begin{array}{l}
a=\sqrt{t} \cdot \frac{1}{\cos \theta}, \frac{b}{\sqrt{2}}=\sqrt{t} \cdot \tan \theta\left(0<\theta<\frac{\pi}{2}, t>0\right).
\end{array}
$$

Thus, $f(a, b)=t k(0<t \leqslant 1, k<1)$, and the range of $f(a, b)$ is $f(a, b)<1$, i.e., $f(a, b)$ can take any value in $(-\infty, 1)$ (since $t$ and $k$ are independent).

For the second part:
$$
\begin{array}{l}
\text { (1) }\left\{\begin{array}{l}
k<-2, \\
3+\sqrt{2 k^{2}+2 k+5}>\frac{3}{2} \sqrt{2}-2, \\
3+\sqrt{2 k^{2}+2 k+5}>\sqrt{2}(2+k)
\end{array}\right. \\
\text { or (2) }\left\{\begin{array}{l}
k>-2, \\
\sqrt{2 k^{2}+2 k+5}<3, \\
3-\sqrt{2 k^{2}+2 k+5}<\sqrt{2}(2+k) .
\end{array}\right.
\end{array}
$$
From (1), we get $k<-2$, and from (2), we get $-2<k<1$. Therefore, the range of $k$ is $k<1$.
Since $f(a, b)=t k(0<t \leqslant 1, k<1)$, the range of $f(a, b)$ is $f(a, b)<1$, i.e., $f(a, b)$ can take any value in $(-\infty, 1)$ (since $t$ and $k$ are independent).