Example 5 Find all odd prime numbers $p$ such that $p \mid \sum_{k=1}^{103} k^{p-1}$.

 untranslated text remains the same as requested.
If $p>103$, then for $1 \leqslant k \leqslant 103$, we have
$$
\begin{array}{l}
k^{p-1} \equiv 1(\bmod p), \\
\sum_{k=1}^{103} k^{p-1} \equiv 103(\bmod p) .
\end{array}
$$

Therefore, $p \leqslant 103$.
Let $103=p q+r(0 \leqslant r < q)$, then $r=q, 103=p q+r=(p+1) r$. Since 103 is a prime number, we get $p=102, r=1$, which is a contradiction.
If $p \leqslant q$, then
$$
103=p q+r \geqslant p^{2} \Rightarrow p=3,5,7 \text {. }
$$

When $p=3$, $q=34 \equiv 1=r(\bmod 3)$;
When $p=5$, $q=20 \equiv 2(\bmod 3), 2 \neq r=3$;
When $p=7$, $q=14 \equiv 2(\bmod 3), 2 \neq r=5$. Therefore, $p=3$ is the solution.