5. The curve $x^{2}+y^{2}-a y=0$ intersects with $a x^{2}+b x y+x$ $=0$ at exactly 3 distinct points. Therefore, it must be that ( ).
(A) $\left(a^{4}+4 a b+4\right)(a b+1)=0$
(B) $\left(a^{4}-4 a b-4\right)(a b+1)=0$
(C) $\left(a^{4}+4 a b+4\right)(a b-1)=0$
(D) $\left(a^{4}-4 a b-4\right)(a b-1)=0$
5. B.

It is clear that $a \neq 0$. The curve $a x^{2}+b x y+x=0$ consists of the two lines $x=0$ and $a x+b y+1=0$. The line $x=0$ intersects the circle $x^{2}+y^{2}-a y=0$ at two distinct points $(0,0)$ and $(0, a)$. According to the problem, there are two possibilities:
(1) The line $a x+b y+1=0$ is tangent to the circle $x^{2}+\left(y-\frac{a}{2}\right)^{2}=\frac{a^{2}}{4}$ at a third point. In this case, 
$$
\frac{\left|a \cdot 0+b \cdot \frac{a}{2}+1\right|}{\sqrt{a^{2}+b^{2}}}=\frac{a}{2},
$$ 
which simplifies to
$$
a^{4}-4 a b-4=0 \text {; }
$$
(2) The line $a x+b y+1=0$ passes through the point $(0, a)$ and is not parallel to the coordinate axes. In this case, $a b+1=0$.