Prove instability using Lyapunov function Consider the system:
\begin{align}
x' &= x^3 + xy \\
y' &= -y + y^2 + xy - x^3 \\
\end{align}
I want to prove the origin is an unstable point by using the Lyapunov function $V(x,y) = \tfrac{x^4}{4} - \tfrac{y^2}{2}$ (this is a hint provided in the exercise).
In order to use Chetaev instability theorem I would like to prove that there is a domain $U$ in a punctured neighborhood of $0$ such that
$$V'(x,y) = \frac{\partial E}{\partial x} (x^3 + xy) + \frac{\partial E}{\partial y} (-y + y^2 + xy - x^3) = x^6 + x^4y + y^2 - y^3 -xy^2 + x^3y$$
is strictly positive, which Wolfram Alpha confirms (indeed, that function is strictly positive in a disk around $0$).
I have tried to show that $0$ is a local minimum, but the Hessian test is not conclusive as the Hessian is positive semidefinite. I have also tried to bound the expression below by $0$ without any luck.
Could you give me any hint to proceed? Thanks in advance

The dominant term is  $x^6+ x^3 y+ y^2$. This term is positive (complete the square). All other terms are small when $x,y,$ are small, so they can be controlled. Here are the details.
Write
$$ \begin{aligned} V'(x,y) &= x^6 + x^4y + y^2 - y^3 -xy^2 +
x^3y\\ &= x^6 + (1+x)x^3y +(1 - x-y)y^2 \\ & \ge x^6 +(1+x)x^3y + \frac{1}{2}y^2 \quad \text{if} \; |x| + |y| \le \frac{1}{4}\\ &\ge x^6-\frac{5}{4}|x|^3|y| + \frac{1}{2}y^2 \quad \text{since} \; |x| \le \frac{1}{4}\\ &= (|x|^3 - \frac{5}{8}|y|)^2 + \frac{7}{64}y^2 \\ & \ge 0 
\end{aligned} 
$$
with equality iff $x = y = 0$.
