$ \int \frac{\cosh(x)}{ (a^2 \sinh^2(x) + b^2 \cosh^2(x))^{\frac{3}{2}} } dx$ I am tring to obtain following formula [actually I obtained this resut via mathematica]
\begin{align}
 \int  \frac{\cosh(x)}{ (a^2 \sinh^2(x) + b^2 \cosh^2(x))^{\frac{3}{2}} } dx 
= \frac{\sqrt{2} \sinh(x)}{a^2 \sqrt{a^2-b^2 + (a^2+b^2) \cosh(2x)}} + C
\end{align}
Since the answer contains $\cosh(2x)$ 
My first trial was
\begin{align}
a^2 \sinh^2(x) + b^2 \cosh^2(x) = a^2 \cosh(2x) + (b^2 -a^2) \cosh^2(x) 
\end{align}
Seems not good.. 
My second trial is 
\begin{align}
a^2 \sinh^2(x) + b^2 \cosh^2(x)  = (a^2+b^2) \sinh^2(x) + b^2 
\end{align}
and this also does not good for integrand.... 
How one can obtain this integral? 

Use your 2nd trial:
$$
\newcommand{\abs}[1]{\left\vert #1 \right\vert}
\newcommand\rme{\mathrm e}
\newcommand\imu{\mathrm i}
\newcommand\diff{\,\mathrm d}
\DeclareMathOperator\sgn{sgn}
\renewcommand \epsilon \varepsilon
\newcommand\trans{^{\mathsf T}}
\newcommand\F {\mathbb F}
\newcommand\Z{\mathbb Z}
\newcommand\R{\Bbb R}
\newcommand \N {\Bbb N}
$$
\begin{align*}
I &= \int \frac {\diff (\sinh x)}{((a^2+b^2) \sinh^2 x +b^2 )^{3/2}} \\
&= \frac 1{b^2 \sqrt {a^2 +b^2}} \int \frac {\diff \left(\sqrt{\frac {a^2 + b^2} {b^2}} \sinh x\right)} {\left(\frac {a^2 + b^2}{b^2} \sinh^2 x + 1\right)^{\frac{3}{2}}} \\
&=\frac 1{b \sqrt {a^2 + b^2}} \int \frac {\diff (\sinh y)}{(\sinh^2 y + 1)^{\frac{3}{2}}} \\
&= \frac 1{b \sqrt {a^2 + b^2}} \int \cosh (y)^{-2} \diff y\\
&= \frac 1{b \sqrt {a^2 +b ^2}} \int \diff (\tanh y) \\
&= \frac {\tanh y} {b\sqrt {a^2 + b ^2}},
\end{align*}
where $\sinh y =b^{-1} \sqrt {a^2 +b^2}\sinh x$. Hence
$$
\cosh^2 y = \sinh^2 y + 1 = \frac 1{b^2} ((a^2 + b^2) \sinh^2 x + b^2) = \frac 2{b^2} (2b^2 + (a^2 + b^2 ) (\cosh (2x ) - 1)) = \frac 2{b^2} ((a^2 + b^2 ) \cosh (2x) - (a^2 -b^2)), 
$$
and
$$
\tanh^2 y = \frac {\sinh^2 y} {\cosh^2 y} = 1 - \frac 1{\sinh^2 y + 1} \implies \tanh y = \frac {\sqrt {a^2 +b^2} \sinh x} {\sqrt {(a^2+b^2 )\sinh^2 x} + b^2} 
$$
hence
$$
I = \frac {\sinh x} {b \sqrt {(a^2 +b^2) \sinh^2 x + b^2}} = \frac {\sqrt 2 \sinh x} {\sqrt {(a^2+b^2) \cosh(2x) + (b^2 - a^2)}}
$$
as we desire, where $2\sinh^2 (x) = \cosh (2x) - 1$. 
