Example 4 Find all positive integers $n$, such that
$$
323 \mid\left(20^{n}+16^{n}-3^{n}-1\right) \text {. }
$$
Notice that, $323=17 \times 19$.
When $n$ is even, because
$$
\begin{array}{l}
20^{n}+16^{n}-3^{n}-1 \\
\equiv 1+3^{n}-3^{n}-1 \equiv 0(\bmod 19), \\
20^{n}+16^{n}-3^{n}-1 \\
\equiv 3^{n}+1-3^{n}-1 \equiv 0(\bmod 17),
\end{array}
$$

thus, equation (1) holds;
When $n$ is odd,
$$
\begin{array}{l}
20^{n}+16^{n}-3^{n}-1 \equiv 3^{n}-1-3^{n}-1 \\
\equiv-2 \not \equiv 0(\bmod 17),
\end{array}
$$

at this point, $323 \nmid\left(20^{n}+16^{n}-3^{n}-1\right)$.
Therefore, equation (1) holds if and only if $n$ is even.