Find $x^2+y^2$ if $x^2-\frac{2}{x}=3y^2$ and $y^2-\frac{11}{y}=3x^2$ 
Find $x^2+y^2$ if $x^2-\frac{2}{x}=3y^2$ and $y^2-\frac{11}{y}=3x^2$.

My try:
$$\frac{y^2-\frac{11}{y}}{3}-\frac{2}{x}=3y^2$$
then?

[ EDIT #3 ]  The previously posted answer remains copied below, under Alternative Solution.
Write the equations as:
$$
\begin{cases}
\begin{align}
2 &= x\,\left(x^2-3y^2\right) \\
11 &= y\,\left(y^2-3x^2\right)
\end{align}
\end{cases}
$$
Square the two and add together:
$$
\begin{align}
125 = 2^2 + 11^2 &= x^2\,\left(x^2-3y^2\right)^2 + y^2\,\left(y^2-3x^2\right)^2 \\
 &= x^6 - 6 x^4 y^2 + 9 x^2y^4 \;\;+\;\;y^6 - 6 y^4x^2 + 9 y^2x^4 \\
 & = x^6 + 3 x^4y^2 + 3 x^2y^4+y^6 \\
 &= \left(x^2+y^2\right)^3
\end{align}
$$
Therefore $\;x^2+y^2 = \sqrt[3]{125} = 5\,$.

[ Alternative Solution ]

As far as brute force goes, you had the right idea to try and eliminate one variable between the equations, but it works out better if first bringing one of the equations to contain only even powers.
From the the second equation:
$$x^2 = \frac{y^3-11}{3y} \tag{1}$$
Rearranging and squaring the first equation, then substituting $x^2$ from $(1)$:
$$
\begin{align}
\left(x^2-3y^2\right)^2 & = \frac{2^2}{x^2} \\[5px]
\frac{y^3-11}{3y} \left(\frac{y^3-11}{3y}-3y^2\right)^2 & = 4 \\[5px]
\left(y^3-11\right)\left(8y^3+11\right)^2 & = 108 y^3 \\[5px]
64 y^9 - 528 y^6 - 1923 y^3 - 1331 & = 0 \tag{2}
\end{align}
$$
Equation $(2)$ is a cubic in $t=y^3$ and has the "obvious" root $t=-1$ correspoding to $y=-1\,$. Dividing by $t+1$ leaves a quadratic in $t$ which gives the other two roots $t=\cfrac{37 \pm 30 \sqrt{3}}{8}\,$. Guessing that those might be of the form $(a \pm b\sqrt{3})^3$ and identifying coefficients turns out to work, which gives the corresponding $y = \cfrac{1 \pm 2\sqrt{3}}{2}$.
For each of the three roots $y$ we can calculate $x^2$ from $(1)$ then $x^2+y^2\,$: 
$$
\begin{cases}
\begin{alignat}{3}
y &= -1 \quad\quad & x^2 &= 4 \quad\quad & x^2+y^2 &= 5 \\[5px]
y &= \frac{1 \pm 2\sqrt{3}}{2} \quad\quad & x^2 &= \frac{7 \mp 4 \sqrt{3}}{4} \quad\quad & x^2+y^2 &= 5
\end{alignat}
\end{cases}
$$
(Meta-comment: I didn't see an obvious way to derive $x^2+y^2$ directly, without actually solving the system, though the nice end result strongly suggests that there would be one.)

[ EDIT ] For a computer-assisted direct solution, note that the problem is equivalent to eliminating $x,y$ between the given equations and $z=x^2+y^2\,$, then solving the resulting equation in $z$.

Elimination can be done using polynomial resultants, and Wolfram Alpha gives the resultant as:
$$2^{24}\,\left(z^3-125\right)^6$$
Quite obviously, $z=5$ is the only real root so $x^2+y^2=5\,$.

[ EDIT #2 ]  The following generalization explains the "magic constants" in the given problem:

$$x^2-\frac{a}{x}=3y^2\;, \quad y^2-\frac{b}{y}=3x^2 \quad \implies \quad x^2+y^2 = \sqrt[3]{a^2+b^2}$$
