Application of Poincaré-Bendixson theorem Consider the system
$$x' = 3xy^2-x^2y \\
y' = 5x^2y - xy^2$$
Show that the system has no periodic solutions.
This is a tricky example. Linearization leads nowhere and I'm having a hard time constructing a Lyapunov function that does the trick. $V = 1/2(x^2+y^2)$ gives
$$V'(x,y) = 3x^2y^2-x^3y +5x^2y^2 -xy^3 = 8x^2y^2 -xy(x^2+y^2))$$
But this doesn't tell us much nice things about the origin. If anything, it looks as though the origin is repelling since small perturbations gives us that the $8x^2y^2$ term dominates the minus term. Maybe it's possible to show that there are no elliptical orbits somehow, but that doesn't exclude other, more exotic, periodic trajectories.
How to proceed...?

I'll give the solution to your non-linear system $$\frac{dx}{dt}=3xy^2-x^2y$$
and $$\frac{dy}{dt}=5x^2y-xy^2$$
Divivding eq 2 by eq 1, we get $$\frac{dy}{dx}=\frac{5x^2y-xy^2}{3xy^2-x^2y}$$
Now, dividing by $x^3$ both the numenator and the denumenator we get that $$\frac{dy}{dx}=\frac{5\frac{y}{x}-(\frac{y}{x})^2}{3(\frac{y}{x})^2-\frac{y}{x}}$$
Let $y=x \cdot v$ then $\frac{dy}{dx}=x\frac{dv}{dx}+v$, thus the D.E transformed into a separable eq.
$$x\frac{dv}{dx}+v=\frac{5v-v^2}{3v^2-v}=\frac{5-v}{3v-1},$$ therefore
$$x\frac{dv}{dx}=\frac{5-v-3v^2+v}{3v-1}=\frac{5-3v^2}{3v-1}$$
$$ \Rightarrow \frac{3v-1}{5-3v^2}dv=\frac{dx}{x}$$ 
$$ \Rightarrow \int{\frac{3v-1}{5-3v^2}dv}=\int{\frac{dx}{x}}$$ 
Setting $
\sqrt {\frac{3}{5}} v= \sin \theta   \Rightarrow \sqrt {\frac{5}{3}} \cos \theta d\theta  = dv$, then 
$$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int{\frac{dx}{x}}$$ 
so that $$\int{\frac{3\sqrt {\frac{5}{3}} \sin \theta 
-1}{5(1- \sin^2 \theta) }\sqrt {\frac{5}{3}} \cos \theta d\theta}=\int {\tan \theta d\theta }  - \sqrt {\frac{5}{3}} \frac{1}{5}\int {\sec \theta d\theta }  
\\
=  - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
$$
Hence the solution is, $$
 - \ln \left| {\cos \theta } \right| - \frac{1}{{\sqrt {15} }}\ln \left| {\sec \theta  + \tan \theta } \right|
=\ln \left| {x } \right|  +C$$ Finally, we may write the result as
$$\left| {\cos \theta } \right|\cdot
\left| {\sec \theta  + \tan \theta } \right|^{{\textstyle{1 \over {\sqrt {15} }}}} 
=\frac{k}{x}$$ where $k=\exp(-C)$.
