Nested equilateral triangles Let triangle $ABC$ is an equilateral triangle. Triangle $DEF$ is also an equilateral triangle and it is inscribed in triangle $ABC \left(D\in BC,E\in AC,F\in AB\right)$. Find $\cos\measuredangle DEC$ if $AB:DF=8:5$.

Firstly, I would be very grateful if someone can explain to me how I am supposed to draw the diagram. Obviously I have made it by sight.
Let $\measuredangle DEC=\alpha$. We can note that $\triangle AEF \cong \triangle BFD \cong CDE$. This is something we can always use in such configuration. So $$AE=BF=CD, $$ $$AF=BD=CE.$$ Let $AB=BC=AC=8x$ and $DF=DE=EF=5x$. If we denote $CD=y,$ then $CE=AC-AE=AC-CD=8x-y$. Cosine rule on $CED$ gives $$25x^2=(8x-y)^2+y^2-2y\cos60^\circ(8x-y)$$ which is a homogenous equation. I got that $\dfrac{y}{x}=4\pm\sqrt{3}.$ Now using the sine rule on $CED$ $$\dfrac{CD}{DE}=\dfrac{\sin\alpha}{\sin60^\circ}\Rightarrow \sin\alpha=\dfrac{\sqrt{3}}{10}\cdot\dfrac{y}{x}=\dfrac{4\sqrt3\pm3}{10}.$$ Now we can use the trig identity $\sin^2x+\cos^2x=1$ but it doesn't seem very rational. Can you give me a hint? I was able to find $\sin\measuredangle DEC$ in acceptable way, but I can't find $\cos\measuredangle DEC$...

As others have done, I'll assume $AB=8,DF=5$ without loss of generality. Then equilateral triangles $\triangle ABC$ and $\triangle DEF$ have area $(\sqrt{3}/4)AF^2 = 16\sqrt{3}$ and $(\sqrt{3}/4)DF^2=(25/4)\sqrt{3}$ respectively. Since the triangles  $\triangle FAE,\triangle DBF, \triangle ECD$ are all congruent, they must each have area $$\frac{1}{3}\left(16\sqrt{3}-\frac{25}{4}\sqrt{3}\right)=\frac{13\sqrt{3}}{4}.$$ But each triangle has a 60-degree base angle, so we can also write the area of $\triangle FAE$ as
$$\frac12 AF\cdot AE \cdot \sin 60^\circ$$
Since $AE=AC-EC=8-AF$, this becomes $$\frac12 AF(8-AF)\frac{\sqrt{3}}{2}= \frac{13\sqrt{3}}{4}\implies AF(8-AF)=13$$
with $x=4\pm \sqrt{3}$ as solutions. As a minor variation on the other approaches, we may then use the law of cosines on triangle $\triangle CED$ to write
$$CD^2=CE^2+ED^2-2CE\cdot ED\cos\measuredangle DEC$$
and therefore
\begin{align}
\cos\measuredangle DEC
&=\frac{CE^2+ED^2-CD^2}{2CE\cdot ED}\\
&=\frac{(4\pm \sqrt{3})^2+5^2-(4\mp \sqrt{3})^2}{2(5)(4\mp \sqrt{3})}\cdot \frac{4\mp \sqrt{3}}{4\mp \sqrt{3}}\\
&=\frac{(25\pm 16\sqrt{3})(4\mp \sqrt{3})}{130}\\
&=\frac{52\pm 39\sqrt{3}}{130}\\
&=\frac{4\pm 3\sqrt{3}}{10}
\end{align}
as others have obtained.
