Simple Integral $\int_0^\infty (1-x\cot^{-1} x)dx=\frac{\pi}{4}$. Hellow I am trying to prove this result.
$$
I:=\int_0^\infty (1-x\cot^{-1} x)dx=\frac{\pi}{4}.
$$
The indefinite integral exists for this integral.
The function $\cot^{-1} x$ is the arc-cotangent function, not the multiplicative inverse.  Note, we can not just break the integral up into two pieces because we will have problems with divergence.  
Using the relation 
$$
x\cot^{-1} x=x \tan^{-1} \frac{1}{x},\to \quad  I=\int_0^\infty \left(1-x\tan^{-1} \frac{1}{x}\right)dx
$$
may be of help to some but didn't help me.  I am not sure if I am missing a clever substitution, perhaps integration by parts will work, I obtained using this method
$$
I=(x-x^2\cot^{-1} x)\big|^\infty_0-\int_0^\infty \frac{x^2}{x^2+1}dx+\int_0^\infty x\cot^{-1} x \, dx
$$
but this is clearly a problem since we have divergence issue now.
The indefinite integral is given by
$$
\int (1-x\cot^{-1} x ) dx =\frac{1}{2}\left(x+\tan^{-1} x-x^2 \cot^{-1} x\right)
$$
but I am unable to prove this result.  Thanks.  

Integrating by parts (as you've done),
$$\begin{align}\int x \cot^{-1}(x) \ dx &= x^{2} \cot^{-1} (x) - \int x\cot^{-1}(x) \ dx + \int \frac{x^{2}}{1+x^{2}} \ dx\\
\implies \int x \cot^{-1}(x) \ dx &= \frac{1}{2} \Big(x^{2} \cot^{-1} (x) + \int \ dx - \int \frac{1}{1+x^{2}} \ dx\Big)\\
&= \frac{1}{2} \Big(x^{2} \cot^{-1} (x) + x - \tan^{-1}(x)\Big).\end{align}$$
Therefore,
$$\begin{align}\int \left( 1- x \cot^{-1}(x) \right) \ dx &= x - \frac{1}{2} \Big(x^{2} \cot^{-1} (x) + x - \tan^{-1}(x)\Big)\\
&= \frac{1}{2} \left(x + \tan^{-1}(x) - x^{2}\cot^{-1}{x} \right).\end{align}$$
EDIT:
And just to add,
$$ \lim_{x \to \infty} \left(x + \tan^{-1}(x) - x^{2}\cot^{-1}(x) \right) = \lim_{x \to \infty} \left[x + \tan^{-1}(x) - x^{2} \left(\frac{1}{x} + \mathcal{O}(x^{-3}) \right) \right] = \frac{\pi}{2}.$$
