$\frac{x}{y}-\frac{y}{x}=\frac56$ and $x^2-y^2=5$ 
Solve the system: $$\begin{array}{|l}
 \dfrac{x}{y}-\dfrac{y}{x}=\dfrac{5}{6} \\ x^2-y^2=5 \end{array}$$

First, we have $x,y \ne 0$. Let's write the first equation as:
$$\dfrac{x}{y}-\dfrac{y}{x}=\dfrac{5}{6} \Leftrightarrow \dfrac{x^2-y^2}{xy}=\dfrac{5}{6}$$
We have $x^2-y^2=5$, therefore $xy=6$. What to do next?

HINT
From the trigonometric point of view, one can substitute $x = r\cos(\theta)$ and $y = r\sin(\theta)$, from whence we get
\begin{align*}
\begin{cases}
r^{2}\cos(2\theta) = 5\\\\
r^{2}\sin(2\theta) = 12
\end{cases} \Longrightarrow \frac{25}{r^{4}} + \frac{144}{r^{4}} = 1 \Longrightarrow r^{4} = 169 \Longrightarrow r = \sqrt{13}
\end{align*}
since $r \geq 0$. Can you take it from here?
