Calculate $\iint\frac{dxdy}{(1+x^2+y^2)^2}$ over a triangle 
Calculate
$$\iint\frac{dxdy}{(1+x^2+y^2)^2}$$ over the triangle $(0,0)$, $(2,0)$, $(1,\sqrt{3})$.    

So I tried changing to polar coordinates and I know that the angle is between $0$ and $\frac{\pi}{3}$ but I couldn't figure how to set the radius because it depends on the angle.

Here is an alternative to Robert's nice way to find how $r(\theta)$ depends on $\theta$.
Let $A(0,0)$, $B(2,0)$ and $C(1,\sqrt{3})$ be the three vertices of the triangle. Imagin, or see the picture below, a ray starting from $A$ and intersects with the side $CB$ at $D$. Suppose the angle $\angle DAB=\theta$. You want to find the length of $AD$ in terms of $\theta$. You can apply the law of sines here to the triangle $ABD$:
$$
\frac{\sin(\pi-\theta-\pi/3)}{2}=\frac{\sin (\pi/3)}{f(\theta)},
$$
Since $\sin(\pi-a)=\sin(a)$, and $\sin(\pi/2) = \sqrt{3}/2$, it follows that
$$
f(\theta) = \frac{\sqrt{3}}{\sin(\theta+\pi/3)}\;.
$$
Let us continue the calculations that are done in Robert's answer:
$$\iint_T\frac{dxdy}{(1+x^2+y^2)^2}
=\int_{0}^{\pi/3}\left(\int_{0}^{f(\theta)}\frac{\rho d\rho}{(1+\rho^2)^2}\right)\;d\theta
=-\frac{1}{2}\int_{0}^{\pi/3}
\left[
\frac{1}{1+\rho^2 }
\right]_{\rho=0}^{\rho=f(\theta)}\,d\theta=:\frac12 I\;.
$$
where
$$
I=-\int_{0}^{\pi/3}
\frac{1}{1+f^2(\theta) }-1\,d\theta
=\int_{0}^{\pi/3}
\frac{f^2(\theta)}{1+f^2(\theta) }\,d\theta
=\int_{0}^{\pi/3}\frac{3}{3+\sin^2(\theta+\pi/3)}\;d\theta=:3J\;.
$$
Up to this point, you can go directly to the general method of Weierstrass substitution. But in this specific case, some trig substitutions makes the integral easier. 
Observe that $\cos(\pi/2-a)=\sin(a)$. So
$$
\begin{align}
J &= \int_{0}^{\pi / 3} \frac{1}{\cos ^{2}\left(\frac{\pi}{6}-x\right)+3} dx
= \int_{-\pi / 6}^{\pi / 6} \frac{1}{\cos ^{2}(u)+3} du
= \int_{-\pi / 6}^{\pi / 6} \frac{\sec ^{2}(u)}{3 \sec ^{2}(u)+1} du\\
&= \int_{-\pi / 6}^{\pi / 6} \frac{\sec ^{2}(u)}{3 \tan ^{2}(u)+4} du
\quad (\sec^2u = \tan^2u+1)\\
&= \int_{-1 / \sqrt{3}}^{1 / \sqrt{3}} \frac{1}{3 s^{2}+4} ds
\quad (d(\tan u)=\sec^2u\;du)\\
&= \frac{1}{4} \int_{-1 / \sqrt{3}}^{1 / \sqrt{3}} \frac{1}{\frac{3 s^{2}}{4}+1} ds
 =\frac{\sqrt{3}}{6} \int_{-1 / 2}^{1 / 2} \frac{1}{p^{2}+1} d p\\
&=\frac{\sqrt{3}}{3} \tan ^{-1}\left(\frac{1}{2}\right)
= \frac{\sqrt{3}}{3} \cot ^{-1}(2)\;.
\end{align}
$$
So the result is
$$
\frac32J = \frac{\sqrt{3}}{2} \cot ^{-1}(2)\;.
$$

