How to calculate this hard integral $\int_0^{\infty} \frac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx$? How to prove that $\displaystyle \int_0^{\infty} \dfrac{\arctan(x)\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}}\,dx= \frac{1}{4}\pi^2\sqrt{2}-\sqrt{2}\ln^2\left(1+\sqrt{2}\right)$ ?
It's very difficult and I have no idea.

This can be reduced to a class of integral related to polylogarithm, and is related to Legendre-chi function $\chi_2$. The formula
$$\int_{0}^{\frac{\pi}{2}} \arctan(r \sin\theta) \, d\theta
= 2 \chi_{2} \left( \frac{\sqrt{1+r^{2}} - 1}{r} \right) $$
gives $$\tag{1}\int_{0}^{\frac{\pi}{2}} \arctan( \sin\theta) \ d\theta
= 2 \chi_{2} \left( \sqrt{2}-1 \right) = \frac{\pi^2}{8} - \frac{1}{2}\log^2(\sqrt{2}+1)$$
where we used a famous special value of $\chi_2$. For a "quick introduction" about $\chi_2$, see Sangchul Lee's here.

Return to your problem, denote your integral by $I$, note that
$$\int \frac{\sqrt{\sqrt{1+x^2}-1}}{x\sqrt{1+x^2}} dx = \sqrt 2 \arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}}  - 1} }}{{\sqrt 2 }}} \right]$$
so integration by parts  gives
$$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\int_0^\infty  {\arctan \left[ {\frac{{\sqrt {\sqrt {1 + {x^2}}  - 1} }}{{\sqrt 2 }}} \right]} \frac{{dx}}{{1 + {x^2}}}$$ 
making $x\mapsto \tan x$ into the last integral gives
$$I = \frac{\sqrt{2}\pi^2}{4}-\sqrt{2}\underbrace{\int_0^{\pi/2}  {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)dx}}_J $$

Denote $$F(r)=\int_0^{\pi/2}  {\arctan \left( {\frac{{a\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)} dx$$
Then compute $F'(r)$ and do an indefinite integral gives
$$\begin{aligned} F'(r) &= \int_0^{\pi /2} {\frac{{\sqrt {\cos x} \sin (\frac{x}{2})}}{{\cos x + {a^2}\sin {{(\frac{x}{2})}^2}}}} dx \\
& =  - \left[ {\frac{{2a\arctan \left[ {\frac{{a\cos (\frac{x}{2})}}{{\sqrt {1 - {a^2}} \sqrt {\cos x} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 }}{{2 - {a^2}}}\ln \left[ {\sqrt 2 \cos (\frac{x}{2}) + \sqrt {\cos x} } \right]} \right]_0^{\pi /2} \\ &=  - \frac{{\pi a}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2a\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }} + \frac{{2\sqrt 2 \ln (\sqrt 2  + 1)}}{{2 - {a^2}}}\end{aligned}$$
Then to get $J$, we integrate the whole expression with respect to $a$ from $0$ to $1$, only the second integral deserves attention, applying $a\mapsto \sin x$:
$$\begin{aligned}\int_0^1 {\frac{{\arctan \left[ {\frac{a}{{\sqrt {1 - {a^2}} }}} \right]}}{{(2 - {a^2})\sqrt {1 - {a^2}} }}da}  &= \int_0^{\pi /2} {\frac{{x\cos x}}{{2 - {{\sin }^2}x}}dx}  \\&=- \int_0^{\frac{\pi }{2}} {xd\left[ {\arctan (\cos x)} \right]}  \\&= \int_0^{\frac{\pi }{2}} {\arctan (\cos x)dx} \end{aligned}$$
Then use $(1)$. Combining all above evaluation should give
$$J=\int_0^{\pi/2}  {\arctan \left( {\frac{{\sin \frac{x}{2}}}{{\sqrt {\cos } x}}} \right)}dx=\ln^2(\sqrt{2}+1)$$ this proves your result. 
Probably there is some smart way to reduce $J$ directly into $(1)$, I hope someone can point on how.
