5. In $\triangle A B C$, it is known that $\tan ^{2} B=\tan A \cdot \tan C$. Then the range of $\angle B$ is ( ).
(A) $\left[\frac{\pi}{3}, \frac{\pi}{2}\right)$
(B) $\left(\frac{\pi}{2}, \frac{2 \pi}{3}\right]$
(C) $\left(0, \frac{\pi}{3}\right] \cup\left(\frac{2 \pi}{3}, \pi\right)$
(D) $\left[\frac{\pi}{3}, \frac{\pi}{2}\right) \cup\left(\frac{\pi}{2}, \frac{2 \pi}{3}\right)$
5.A.

From $\tan A \cdot \tan C=\tan ^{2} B$,
we get $\tan B=\frac{\tan A+\tan C}{\tan A \cdot \tan C-1}=\frac{\tan A+\tan C}{\tan ^{2} B-1}$.
Thus, $\tan A+\tan C=\tan B\left(\tan ^{2} B-1\right)$.
(2) ${ }^{2}-4 \times$ (1) gives
$$
\begin{array}{l}
0 \leqslant(\tan A-\tan C)^{2} \\
=\tan ^{2} B\left(\tan ^{2} B-1\right)^{2}-4 \tan ^{2} B \\
=\tan ^{2} B\left(\tan ^{2} B+1\right)\left(\tan ^{2} B-3\right) .
\end{array}
$$

Clearly, $\tan ^{2} B\left(\tan ^{2} B+1\right)>0$, so $\tan ^{2} B \geqslant 3$.
Therefore, $\tan B \geqslant \sqrt{3}$ or $\tan B \leqslant-\sqrt{3}$.
Thus, $\angle B \in\left[\frac{\pi}{3}, \frac{\pi}{2}\right)$ or $\angle B \in\left(\frac{\pi}{2}, \frac{2 \pi}{3}\right]$.
Next, we show that $\angle B$ cannot be an obtuse angle.
If $\angle B>\frac{\pi}{2}$, then
$0<\angle A<\pi-\angle B<\frac{\pi}{2}, 0<\angle C<\pi-\angle B<\frac{\pi}{2}$.
Hence, $0<\tan A<\tan (\pi-\angle B)$,
$0<\tan C<\tan (\pi-\angle B)$.
Thus, $\tan A \cdot \tan C<\tan ^{2}(\pi-\angle B)=\tan ^{2} B$.
This contradicts (1), so $\angle B$ cannot be an obtuse angle.
Furthermore, when $\tan B=\sqrt{3}$, we must have
$\tan A-\tan C=0$.
From (1), we get
$\tan A=\tan C=\tan B$.
Therefore, $\angle A=\angle B=\angle C=\frac{\pi}{3}$.
Hence, $B \in\left[\frac{\pi}{3}, \frac{\pi}{2}\right)$.