Integral using polar coordinates Let $X=\{(x,y)\in\mathbf{R}^2\mid x^2+y^2\leqslant 1,x,y\geqslant 0 \}$. Calculate $\int_X xye^{x^2+y^2}\,dx\,dy$. 
By using polar coordinates, I get $=\int_0^{\pi/2}\int_0^1e \cos\theta\sin\theta\,dr\,d\theta=\int_0^{\pi/2}\frac{1}{2}e\sin 2\theta\,d\theta=\left.-\frac{1}{4}e\cos 2\theta\right\vert_0^{\pi/2}=\frac{1}{4}e+\frac{1}{4}e=\frac{1}{2}e $.
By rewrting, I get
$=\int_0^1 \int_{0}^{\sqrt{1-y^2}} xye^{x^2+y^2}\,dx\,dy=\int_0^1 ye^{y^2}\left(\frac{1}{2}e^{x^2}\right)_0^{\sqrt{1-y^2}}\,dy=\int_0^1 \frac{1}{2}ey-\frac{1}{2}ye^{y^2}\,dy=\left.\frac{1}{4}ey^2-\frac{1}{4}e^{y^2}\right\vert_0^1=\frac{1}{4}e-\frac{1}{4}e+\frac{1}{4}e=\frac{1}{4}e$.
The code in Mathematica
f[x] := x*y*Exp[x^2 + y^2] 

Integrate[f[x] Boole[x^2 + y^2 < 1], {x, 0, 1}, {y, 0, 1}]

gives me $\frac{1}{4}$.
Who can tell me which of these three answers is correct and where are my mistakes?

Your polar integral is wrong; it should be
$$\int_0^{\pi/2}\int_0^1r^2\cos\theta\sin\theta\cdot e^{r^2}r\,dr\,d\theta$$
where the $r^2\cos\theta\sin\theta$ comes from rewriting $xy$ and the extra $r$ factor is the Jacobian of the transformation to polar coordinates. Continuing the evaluation, we get
$$=\int_0^{\pi/2}\cos\theta\sin\theta\int_0^1r^3e^{r^2}\,dr\,d\theta$$
$$=\int_0^{\pi/2}\cos\theta\sin\theta[(r^2-1)e^{r^2}/2]_0^1\,d\theta$$
$$=\int_0^{\pi/2}\frac12\cos\theta\sin\theta\,d\theta$$
$$=\frac14\int_0^{\pi/2}\sin2\theta\,d\theta$$
$$=\frac14[-1/2\cdot\cos2\theta]_0^{\pi/2}=\frac14$$
agreeing with Mathematica.
