Eliminating $\theta$ from the system $x\sin\theta-y\cos\theta=-\sin4\theta$, $x\cos\theta+y\sin\theta=\frac52-\frac32\cos4\theta$ 
Eliminate $\theta$ from the system of equations.
  $$\begin{align}
x\sin\theta-y\cos\theta&=\phantom{\frac52\frac32}-\sin4\theta \\
x\cos\theta+y\sin\theta&=\frac52-\frac32\cos4\theta
\end{align}$$

I am stuck at this question after squaring and adding.

My believe that one of the coefficients of $x,y$  in Eliminating $\theta$ from trigonometric system will be negative has been corroborated by the current question.
Solving for $x,y$
$$\dfrac x{-\cos t(3\cos4t-5)-2\sin t\sin4t}=\dfrac y{2\cos t\sin4t-\sin t(3\cos4t-5)}=\dfrac12$$
$$4x=10\cos t-5\cos3t-\cos5t\iff x=5c-4c^5$$
$$4y=-\sin5t+5\sin3t+10\sin t\iff y=5s-4s^5$$  where $c=\cos t,s=\sin t$ using this and this
$$x^2+y^2=25-40(1-2c^2s^2)+16(1-3c^2s^2)$$
$$=1+32c^2s^2=1+8\sin^22t=1+4(1-\cos4t)=5-4\cos4t$$
$$\cos4t=\dfrac{5-x^2-y^2}4\ \ \ \  (1)$$
Again squaring & adding  $$4(x^2+y^2)=4\sin^24t+(5-3\cos4t)^2=25+4(1-\cos^24t)+9\cos^24t-30\cos4t\ \ \ \  (2)$$
Replace the value of $\cos4t$ in $(2)$ from $(1)$
