Example 7 Find all positive integers $n$, such that there exists an integer-coefficient polynomial $P(x)$, satisfying $P(d)=\left(\frac{n}{d}\right)^{2}$, where, $d$ is each divisor of $n$. [3]
For a polynomial $P(x)$ with integer coefficients, and positive integers $a, b (a \neq b)$, we always have
$$
(a-b) \mid (P(a)-P(b)).
$$

This is an invariant.
If $n=1$, then $P(1)=1$. Hence, we can take the polynomial $P(x)=x$.

If $n$ is a prime number, then $n$ has only two factors, 1 and $n$. The polynomial $P(x)$ must satisfy
$$
P(1)=n^{2}, P(n)=1.
$$

Let $P(x)=a x+b$.
From $P(1)=n^{2}, P(n)=1$, we get
$$
P(x)=-(n+1) x+n^{2}+n+1.
$$

If $n=k l(1<k \leqslant l)$ is a composite number, then
$$
\begin{array}{l}
P(1)=n^{2}, P(l)=k^{2}, P(k)=l^{2}, P(n)=1. \\
\text { By }(n-k) \mid (P(n)-P(k)) \\
\Rightarrow k(l-1) \mid (1-l)(1+l) \\
\Rightarrow k \mid (1+l).
\end{array}
$$

Similarly, by
$$
(n-l) \mid (P(n)-P(l)) \Rightarrow l \mid (1+k).
$$

Thus, $k l \mid (1+k)(1+l) \Rightarrow k l \mid (1+k+l)$
$$
\Rightarrow k l \leqslant 1+k+l \Rightarrow (k-1)(l-1) \leqslant 2.
$$

Solving this, we get $k=2, l=3$, i.e., $n=6$.
Then $P(1)=36, P(2)=9, P(3)=4, P(6)=1$.
Let $Q(x)=P(x)-1$.
Then $Q(1)=35, Q(2)=8, Q(3)=3, Q(6)=0$.
Let $Q(x)=(x-6) R(x)$. Then
$$
R(1)=-7, R(2)=-2, R(3)=-1.
$$

Let $R(x)=-1+(x-3) S(x)$. Then
$$
S(1)=3, S(2)=1.
$$

Thus, we find $S(x)=5-2 x$.
Therefore, $P(x)=1+(x-6)[-1+(x-3)(5-2 x)]$.