Finding $\cos ( 2 \sin^{-1}( \frac{5}{ 13} )) $ The following problem is from the $8$th edition of the book Calculus, by James Stewart. It is problem number $9$ in section $6.6$.
Problem:
Find an exact value for the expression:
$$ \cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } $$
Answer:
\begin{align*}
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=
 \sqrt{  1 - \sin^2{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } } \\
%
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=
 \sqrt{  1 - 2 \sin^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right)  } \cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right)  } } \\
%
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=
 \sqrt{  1 - 2 \left( \frac{25}{13^2} \right)  \cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right)  } } \\
%
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=
 \sqrt{  1 - \left( \frac{50}{13^2} \right)  \cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right)  } } \\
\end{align*}
\begin{align*}
\cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right) } &=
 1 - \sin^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right) } \\
\cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right) } &= 1 - \frac{25}{169} = \frac{169 - 25}{169} \\
\cos^2{\left(  \sin^{-1}\left( \frac{5}{13} \right) \right) } &= \frac{144}{169} \\
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=
 \sqrt{  1 - \left( \frac{50}{13^2} \right)  \left(  \frac{144}{169} \right) } \\
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &= \sqrt{  \frac{13^4 - 50(144)}{13^4}  } \\
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &= \sqrt{  \frac{21361}{13^4}  } \\
\cos{\left(  2 \sin^{-1}\left( \frac{5}{13} \right) \right)  } &=  \frac{  \sqrt{ 21361 } } { 169 }
\end{align*}
The book's answer is $\frac{119}{169}$ and SciLab matches the book. Where did I go wrong?

Your mistake is when you squared $\sin2\theta$, it should be $4\sin^{2}\theta\cos^{2}\theta$ not $2\sin^{2}\theta\cos^{2}\theta$, but in case you want an easier method than going for a long method, here is one:
Just let $\displaystyle \theta=\sin^{-1}\left(\frac{5}{13}\right)$ so that we have $\sin \theta=\dfrac{5}{13}$ and now we know that $\displaystyle \cos 2\theta=1-2\sin^{2}\theta=1-2\left(\frac{5}{13}\right)^2=\frac{119}{169}.$
