How can I find the maximum and minimum of $f(x,y)$ in $D=\{ (x,y) \in \Bbb{R}^2 \mid x^2+y^2 \le 4 \}$? How can I find the maximum and minimum of $f(x,y)$ in $D=\{ (x,y) \in \Bbb{R}^2 \mid x^2+y^2 \le 4 \}$
where 
$$f(x,y)=x^4+y^4-x^2-2xy-y^2$$
Answer:
$$f_x=4x^3-2x-2y$$
$$f_y=4y^3-2y-2x$$
Critical points are $(0,0), (1,1), (-1,-1)$. 
All of these critical points are in D.
$$f(0,0)=0$$
$$f(1,1)=-2$$
$$f(-1,-1)=-2$$
Now we can find the extreme values of $f$ on the boundary of the region $x^2+y^2=4$
$$x^2=4-y^2$$
Therefore we have,
$$g(y)=2y^4-8y^2-2y \sqrt{4-y^2}+16$$
$$g'(y)=8y^3-16y-2 \sqrt{4-y^2}+2y^2(4-y^2)^{-1/2}=0$$
$\color{red}{problem:}$
I could do only up to here. I don't know how to solve the above equation to get the values for $y$.
After finding extreme values of this, we can compare those values with the above and we can find the maximum and minimum values.

Probably easier to write the boundary as
$$x=2\cos\theta\ ,\quad y=2\sin\theta\ .$$
Then
$$\eqalign{f(x,y)
  &=16\cos^4\theta+16\sin^4\theta-4\cos^2\theta-8\cos\theta\sin\theta-4\sin^2\theta\cr
  &=4(1+\cos2\theta)^2+4(1-\cos2\theta)^2-4\sin2\theta-4\cr
  &=4+8\cos^22\theta-4\sin2\theta\cr
  &=12-4\sin2\theta-8\sin^22\theta\cr}$$
and it should not be hard to find extreme values of this.
