Evaluate the definite integral $\int_0^{2\pi}\arctan\left((a\cos x + b\sin x)^2\right)dx$ Writing:
Integrate[ArcTan[(a Cos[x] + b Sin[x])^2], {x, 0, 2 Pi}, Assumptions -> a^2 + b^2 > 0]


$$\int_0^{2\pi}\arctan\left((a\cos x + b\sin x)^2\right)dx,$$
  where $a $ and $b $ are real numbers.

I get:
2 Pi ArcTan[Sqrt[1/2 (-1 + Sqrt[1 + (a^2 + b^2)^2])]]


$$2\pi\arctan\sqrt{\frac{\sqrt{1 + (a^2 + b^2)^2}-1}2} $$

How to derive this result on paper?

Denote $$C := a^2 + b^2 .$$ Then, we can find an angle $x_0$ such that $a = \sqrt{C} \cos x_0$ and $b = -\sqrt{C} \sin x_0$. The angle sum formula for $\sin$ lets us rewrite the quantity in the inner parentheses of the integrand as $$a \sin x + b \cos x = \sqrt{C} \sin (x - x_0).$$ Then, appealing to the periodicity of the integrand lets us rewrite the integral as
$$I(C) := \int_0^{2 \pi} \arctan (C \sin^2 x) \,dx .$$
Differentiating under the integral sign gives
$$I'(C) = \int_0^{2 \pi} \frac{\sin^2 x\, dx}{1 + C^2 \sin^4 x} .$$
Now, use symmetry to rewrite $I'(C)$ in terms of an integral over $[0, \pi]$, and apply the Euler substitution $x = 2 \arctan t, \,dx = \frac{2\,dt}{1 + t^2}$, giving the rational integral $$I'(C) = 16 \int_0^{\infty} \frac{t^2 (t^2 + 1) \,dt}{(t^2 + 1)^4 + 16 C^4 t^4}.$$
Integrating gives
$$I'(C) = \frac{\pi \sqrt{2}}{\sqrt{1 + \sqrt{1 + C^2}} \sqrt{1 + C^2}} .$$ (This can be done with contour integration, which in this case is tedious but straightforward. Quite possibly there is a better method, and I would be grateful to learn it.)
Since $I(0) = 0$, we have
$$I(C) = \pi \sqrt{2} \int_0^C \frac{dc}{\sqrt{1 + \sqrt{1 + c^2}} \sqrt{1 + c^2}} = 2 \pi \sqrt{2} \int_0^{u_0} \frac{du}{u^2 + 2} ;$$ the latter equality follows from applying the substitution $c^2 + 1 = (u^2 + 1)^2$, and $u_0$ is the $u$-value corresponding to $c = C$. The integral on the right-hand side is elementary, and so one can produce an explicit formula for $I(C)$ in terms of $C$ and hence in terms of $a, b$.
