Example 4 Find all pairs of positive integers $(m, n)$ such that $m^{2}-4 n$ and $n^{2}-4 m$ are both perfect squares.
By symmetry, assume $m \leqslant n$.
(1) If $m \leqslant n-1$, then $n^{2}-4 m \geqslant n^{2}-4 n+4=(n-2)^{2}$.
Also, $n^{2}-4 m \leqslant(n-1)^{2}=n^{2}-2 n+1$, and the
equality does not hold, so
$$
n^{2}-4 m=(n-2)^{2} \Rightarrow n=m+1 \text {. }
$$

Thus, $m^{2}-4 n=(m-2)^{2}-8=t^{2}\left(t \in \mathbf{N}_{+}\right)$.
Therefore, $(m+t-2)(m-t-2)=8$.
Since $m+t-2$ and $m-t-2$ have the same parity, and $m+t-2>m-t-2$, we have,
$$
m+t-2=4, m-t-2=2 \text {. }
$$

Solving these, we get $m=5, n=6$ as a solution.
(2) If $m=n$, then
$$
m^{2}-4 n=(m-2)^{2}-4=t^{2}(t \in \mathbf{N}) \text {. }
$$

Solving similarly, we get $m=n=4$.
In summary, all pairs of positive integers that satisfy the conditions are
$$
(m, n)=(4,4),(5,6),(6,5) \text {. }
$$