Problem 5. Let $f(x)=3 x^{2}-7 x-11$. Solve the equation $f(f(x))=x$.

---

The original text has been translated into English, maintaining the original formatting and line breaks.
Answer: $-1, \frac{11}{3}, \frac{3 \pm 4 \sqrt{3}}{3}$.

First solution. A number $x$ satisfies the equation $f(f(x))=x$ if and only if there exists a number $y$ such that the system

$$
\left\{\begin{array}{l}
3 x^{2}-7 x-11=y \\
3 y^{2}-7 y-11=x
\end{array}\right.
$$

is satisfied. Subtracting the second equation from the first, after transformations we get $3(x-y)(x+y-2)=0$, from which either $y=x$, or $y=-x+2$. Substituting $y$ into the first equation of the system. In the first case, we get the equation $3 x^{2}-8 x-11=0$, from which $x=-1$ or $x=\frac{11}{3}$. In the second case, we get the equation $3 x^{2}-6 x-13=0$, from which $x=\frac{6 \pm \sqrt{192}}{2 \cdot 3}=\frac{3 \pm 4 \sqrt{3}}{3}$

Plan of the second solution. We will honestly calculate $f(f(x))-x$:

$\left(3\left(3 x^{2}-7 x-11\right)^{2}-7\left(3 x^{2}-7 x-11\right)-11\right)-x=27 x^{4}-126 x^{3}-72 x^{2}+510 x+429=3\left(9 x^{4}-42 x^{3}-24 x^{2}+170 x+143\right)$, i.e., the original equation is equivalent to

$$
9 x^{4}-42 x^{3}-24 x^{2}+170 x+143=0
$$

Now we can find two rational roots of this equation using the Rational Root Theorem: if $p / q, (p, q)=1$ is a root of a polynomial with integer coefficients, then $p$ is a divisor of the constant term, and $q$ is a divisor of the leading coefficient. We get:

$$
9 x^{4}-42 x^{3}-24 x^{2}+170 x+143=(x+1)(3 x-11)\left(3 x^{2}-6 x-13\right)
$$

The roots of the third factor can be found using the quadratic formula.