Example 2 For a finite set $A$, there exists a function $f: N \rightarrow$ $A$, with the following property: if $i, j \in N$, and $|i-j|$ is a prime number, then $f(i) \neq f(j)$. How many elements must the set $A$ have at least?
Solution: Since the absolute value of the difference between any two numbers among $1,3,6,8$ is a prime number, according to the problem, $f(1)$, $f(3)$, $f(6)$, and $f(8)$ are four distinct elements in $A$, thus $|A| \geqslant 4$.

On the other hand, if we let $A=\{0,1,2,3\}$, and the mapping $f: N \rightarrow A$ is defined as: if $x \in N, x=4 k+r$, then $f(x)=r$, where $k \in N \cup\{0\}, r=0,1,2,3$. For any $x, y \in N$, if $|x-y|$ is a prime number, assume $f(x) = f(y)$, then $x \equiv y(\bmod 4)$, thus $4 \mid x-y\rfloor$, which contradicts the fact that $|x-y|$ is a prime number.
Therefore, $A$ contains at least 4 distinct elements.