5. Find the non-constant real-coefficient polynomials \( P(x) \) and \( Q(x) \) that satisfy
$$
P(Q(x))=P(x) Q(x)-P(x)
$$
$$
\begin{array}{l}
\text { 5. } P(x)=a x^{2}-a(b+1) x+a b, \\
Q(x)=x^{2}-(b+1) x+2 b,
\end{array}
$$

where $a, b \in \mathbf{R}$, and $a \neq 0$.
Let the degrees of $P$ and $Q$ be $m$ and $n$ respectively.
By comparing the degrees on both sides of equation (1), we get
$$
\begin{array}{l}
m n=m+n \\
\Leftrightarrow(m-1)(n-1)=1 \\
\Leftrightarrow m=n=2 .
\end{array}
$$

Thus, $P(x)$ can be expressed as
$$
P(x)=\alpha x^{2}+\beta x+c.
$$

Then equation (1) can be expressed as
$$
\begin{array}{l}
\alpha Q^{2}(x)+\beta Q(x)=P(x) Q(x)-(P(x)+c) \\
\Rightarrow Q(x) \mid(P(x)+c).
\end{array}
$$

Since the degrees of $P$ and $Q$ are the same,
$$
\Rightarrow P(x)+c=a Q(x) \text { (where } a \in \mathbf{R} \text { and } a \neq 0).
$$

Thus, equation (1) can also be expressed as
$$
\begin{array}{l}
a Q(Q(x))-c \\
=(a Q(x)-c) Q(x)-(a Q(x)-c) \\
=a Q^{2}(x)-(a+c) Q(x)+c.
\end{array}
$$

Since $Q(x)$ is not a constant, it can take infinitely many values.
$$
\begin{array}{l}
\text { Then } a Q(x)=a x^{2}-(a+c) x+2 c \\
\Rightarrow Q(x)=x^{2}-(b+1) x+2 b \left(b=\frac{c}{a}\right).
\end{array}
$$

Thus, $P(x)=a Q(x)-a b$
$$
\Rightarrow P(x)=a x^{2}-a(b+1) x+a b.
$$

It is easy to verify that $P$ and $Q$ satisfy equation (1).
Indeed, since $P$ and $Q$ satisfy
$$
P(x)=a Q(x)-a b,
$$

then from equation (1) we have
$$
\begin{array}{l}
a Q(Q(x))-a b \\
=(a Q(x)-a b) Q(x)-(a Q(x)-a b) \\
\Rightarrow Q(Q(x))=Q^{2}(x)+(b+1) Q(x)+2 b.
\end{array}
$$

Clearly, $Q(x)=x^{2}-(b+1) x+2 b$ satisfies.