$k=-\sqrt{3}(\sqrt{5+2\sqrt{6}}+\sqrt{5-2\sqrt{6}})$ 
Let $k$ be equal to:
  $$k=-\sqrt{3} \left(\sqrt{5+2\sqrt{6}}+\sqrt{5-2\sqrt{6}} \right)$$

I am trying to simplify the expression and express $5+2\sqrt{6}$ and $5-2\sqrt{6}$ as squares. I don't think it's smart to multiply the brackets by $-\sqrt{3}$ at first. Is there any algorithm that I should know to express the expressions as squares?

Squaring gives
\begin{eqnarray*}
k&=& -\sqrt{3} \left(\sqrt{5+ 2\sqrt{6}} + \sqrt{5- 2\sqrt{6}}   \right)   \\
k^2&=&3 \left(5+ 2\sqrt{6}+5- 2\sqrt{6} +2\sqrt{ (5+ 2\sqrt{6})( 5- 2\sqrt{6}) }     \right)   \\
k^2&=&3 \left(10 +2\sqrt{ 25- 4 \times{6} }     \right)   \\
k^2&=& 36   \\
\end{eqnarray*}
Now square root this and we recall we expect $k$ to be negative ... so $\color{red}{k=-6}$.
