Example 6 Find all prime numbers $p$ such that
$$
p^{3} \mid \sum_{k=1}^{p-1}\left(\mathrm{C}_{p}^{k}\right)^{2} .
$$
Obviously, $p \neq 2,3$.
When $p \geqslant 5$, note that,
$$
\begin{array}{l}
p-1 \equiv-1(\bmod p), \\
p-2 \equiv-2(\bmod p), \\
\cdots \cdots \\
k \equiv k-p(\bmod p)(k=1,2, \cdots, p) .
\end{array}
$$

Multiplying all the equations, we get
$$
\frac{(p-1)!}{(k-1)!} \pm(p-k)!(\bmod p),
$$

which means $\mathrm{C}_{p-1}^{k-1} \equiv \pm 1(\bmod p)$.
For $k(1 \leqslant k \leqslant p-1)$, there exists $r_{k}\left(1 \leqslant r_{k} \leqslant\right.$ $p-1)$, such that $k r_{k} \equiv 1(\bmod p)$, and when $k$ takes all values from 1 to $p-1$, $r_{k}$ also takes all values from $1,2, \cdots, p-1$.
Also, $\mathrm{C}_{p}^{k}=\frac{p}{k} \mathrm{C}_{p-1}^{k-1}$, so we have
$$
\begin{array}{l}
p^{3}\left|\sum_{k=1}^{p-1}\left(\mathrm{C}_{p}^{k}\right)^{2} \Leftrightarrow p\right| \sum_{k=1}^{p-1} \frac{\left(\mathrm{C}_{p-1}^{k-1}\right)^{2}}{k^{2}} . \\
\text { and } \sum_{k=1}^{p-1} \frac{\left(\mathrm{C}_{p-1}^{k-1}\right)^{2}}{k^{2}} \equiv \sum_{k=1}^{-1} \frac{\left(\mathrm{C}_{p-1}^{k-1}\right)^{2}}{k^{2}}\left(k r_{k}\right)^{2} \\
\equiv \sum_{k=1}^{p-1} r_{k}^{2} \equiv \sum_{k=1}^{p-1} k^{2} \\
\equiv \frac{p(p-1)(2 p-1)}{6}(\bmod p) .
\end{array}
$$

When the prime $p \geqslant 5$, we have
$$
(p-1)(2 p-1) \equiv 1-p^{2} \equiv 0(\bmod 3),
$$

which implies $6 \mid(p-1)(2 p-1)$.
Therefore, $p \left\lvert\, \sum_{k=1}^{p-1} \frac{\left(\mathrm{C}_{p-1}^{k-1}\right)^{2}}{k^{2}}\right.$.
Hence, all primes not less than 5 are the solutions.