Calculate this integral, $\int_0^{\infty}\frac{\ln^2x}{1+x^2}dx$ $\displaystyle\int_0^{\infty}\dfrac{\ln^2x}{1+x^2}dx$
$x=\arctan\alpha$
$dx=\dfrac{1}{1+\alpha^2}d\alpha$
and try a lot items, but didnt arrived anywhere.

Let us evaluate it via the Mellin transform
\begin{equation}
\mathcal{M}[f(x)](s) = \int\limits_{0}^{\infty} x^{s-1} f(x) \mathrm{d} x
\label{eq:1608132}
\tag{2}
\end{equation}
where
\begin{equation}
f(x) = \frac{1}{1+x^{2}}
\label{eq:1608133}
\tag{3}
\end{equation}
Applying the Mellin transform, yields
\begin{equation}
\mathcal{M}[f(x)](s) = \int\limits_{0}^{\infty} \frac{x^{s-1}}{1+x^{2}} \mathrm{d} x = \frac{\pi}{2}\csc\left(\frac{\pi}{2}s\right)
\label{eq:1608134}
\tag{4}
\end{equation}
And thus
\begin{align}
\int\limits_{0}^{\infty} \frac{\mathrm{ln}^{2}(x)}{1+x^{2}} \mathrm{d} x & = \frac{\mathrm{d}^{2}}{\mathrm{d}s^{2}} \int\limits_{0}^{\infty} \frac{x^{s-1}}{1+x^{2}} \mathrm{d} x |_{s=1} \\
& = \frac{\mathrm{d}^{2}}{\mathrm{d}s^{2}} \frac{\pi}{2}\csc\left(\frac{\pi}{2}s\right) |_{s=1} \\
& = \frac{\pi^{3}}{16}[\cos(\pi s) + 3]\csc^{3}\left(\frac{\pi}{2}s\right)  |_{s=1} \\
& = \frac{\pi^{3}}{8}
\label{eq:1608135}
\tag{5}
\end{align}
