Triangle ABC with $\angle{ACB} = 3\angle{ABC }$ and $AB = \frac{10}{3}BC$. Find $\cos{A}\cos{B}\cos{C}$ assume that $\angle{ABC}=y, \angle{ACB}=3y, BC=3x,$ and $AB=10x$ then by sine rule, i obtain following

$ \frac{10x}{\sin{3y}}=\frac{AC}{\sin{y}}=\frac{3x}{\sin{4y}}$

by cosine rule in try to figure out AC

$AC=\sqrt{109x^2-60x^2\cos{y}}$

i have no idea how to combine these two informations in order to solve the problem

From the first equation:
$$10\sin(4y)-3\sin(3y)=0$$
Let's expand everything in terms of $\sin y$ and $\cos y$:
$$\begin{align}\sin(4y)&=\sin(2(2y))\\&=2\sin2y\cos2y\\&=4\sin y\cos y(1-2\sin^2y)\\&=4\sin y\cos y-8\sin^3 y\cos y\\\sin(3y)&=\sin(y+2y)\\&=\sin y\cos2y+\cos y\sin2y\\&=\sin y(1-2\sin^2y)+2\sin y\cos^2y\\&=\sin y-2\sin^3y+2\sin y-2\sin^3y\\&=3\sin y-4\sin^3y\end{align}$$
We then get
$$40\sin y\cos y-80\sin^3y\cos y=9\sin y-12\sin^3y$$
$\sin y=0$ is not a good solution for the problem, so we can divide by it:
$$40\cos y-80(1-\cos^2y)\cos y-9+12(1-\cos^2y)=0$$
Let $\cos y=z$. Then$$40z-80z+80z^3+3-12z^2=0\\80z^3-12z^2-40z+3=0$$
It is not trivial, but from the rational root theorem you get that one root is $z=\frac34$
So the equation becomes $$(4z-3)(20z^2+12z-1)=0$$
Note that $y$ has to be less than $\frac\pi4$, since you have angles $y$, $3y$, and the sum of them has to be less than $\pi$. You can see that $\cos y=\frac34$ is the only acceptable solution. Then the rest should be easy.
