Evaluate the double integral. 
To find:

$$\iint_R(x^2-xy)dA$$

enclosed by

$$y=x, y=3x-x^2,$$
$$x=3x-x^2,$$
$$x=0, x=2,$$
$$y=0, y=2,$$
$$R=((x,y): 0\le x\le 2), (x \le y \le 3x-x^2)$$
$$\int_0^2\int_x^{3x-x^2}(x^2-xy)\,dy\,dx$$
$$\int_0^2 \left. \left(x^2y-x\frac{y^2}{2}\right) \right|_x^{3x-x^2}\,dx$$

where to from here?


With the change of variables $u=x$ and $v=y-x$, that is $x=u$ and $y=v-u$,  we have the Jacobian $\left|\frac{\partial(x,y)}{\partial (u,v)}\right|=1$; from $0\le x\le 2$ we have $0\le u\le 2$ and from $x\le y\le 3x-x^2$, we have $0\le y-x\le 2x-x^2$ we have $0\le v\le 2u-u^2$. The integral becomes
\begin{align}
I&=\int_0^2\int_0^{(2-u)u} (-uv)\,\mathrm dv\,\mathrm du\\
&=\int_0^2 (-u)\,\frac{u^2(2-u)^2}{2} \mathrm du=-\frac{1}{2}\int_0^2(4u^3-4u^4+u^5)\,\mathrm du\\
&=\left.-\frac{1}{2}\left(4u^4-\frac{4u^5}{5}+\frac{u^6}{6}\right)\right|_0^2=-\frac{2^4}{2}\left(4-\frac{4\cdot 2}{5}+\frac{2^2}{6}\right)=-\frac{8}{15}
\end{align}
Check with WolframAlpha
