Nature of the critical point $(0,0)$ of the function $f(x,y)=x^6-2x^2y-x^4y+2y^2$ Consider the function $$f(x,y)=x^6-2x^2y-x^4y+2y^2.$$ The point $(0,0)$ is a critical point. Observe, 
\begin{align*}
f_x & = 6x^5-4xy-4x^3y, f_x(0,0)=0\\ 
f_y & = 2x^2-x^4+4y. f_y(0,0)=0\\
f_{xx} & = 30x^4-4y-12x^2y, f_{xx}(0,0)=0\\
f_{xy} & = 4x-4x^3, f_{xy}(0,0)=0\\
f_{yy} & = 4, f_{yy}=4
\end{align*}
So, in order to determine the nature of the above critical point, we need to check the Hessian at $(0,0)$ which is $0$ and hence the test is inconclusive. $$ H(x,y)= \det \begin{pmatrix} f_{xx} & f_{xy}\\ f_{yx} & f_{yy} \end{pmatrix}=\det \begin{pmatrix} 0 & 0 \\ 0 & 4 \end{pmatrix}=0$$So, I tried to see the function on slices like $y=0$ and $y=x$ but nothing worked. So please suggest me how do I find the nature of the critical point in this case?

You have that
$$
g_a(x)=f(x,ax^2 ) = 2\left( {a^2  - a} \right)x^4  + \left( {1 - a} \right)x^6 
$$
With $0<a<1$ the function $g_a(x)$ has a local maximum. With $a>1$ the function has a local minimum. This means that $(0,0)$ is a saddle point.
