How to find the roots of Martrix Norm? 
Problem:  Compute $||A||$ where $A = \begin{pmatrix} 1 & 1 \\ 0 & 1\end{pmatrix}$


Attempt:

$$
\begin{align*}
\det(A^TA-λI_2) &= 
\det \left( \begin{pmatrix} 1 & 0 \\ 1 & 1\end{pmatrix}
\begin{pmatrix} 1 & 1 \\ 0 & 1\end{pmatrix}
- \begin{pmatrix} λ & 0 \\ 0 & λ\end{pmatrix} \right) \\
&= \det \left( \begin{pmatrix} 1 & 1 \\ 1 & 2\end{pmatrix}
- \begin{pmatrix} λ & 0 \\ 0 & λ\end{pmatrix} \right)  \\
&= \det \left( \begin{pmatrix} 1-λ & 1 \\ 1 & 2-λ\end{pmatrix} \right)  \\
&= (1-λ)(2-λ)-1(1) \\
&= λ^2-3λ+2-1 \\
&= λ^2-3λ+1
\end{align*}$$
The book, skipped the solution and has the answer

"which has roots $λ=\frac{3}{2}\pm\frac{1}{2}\sqrt{5}$. Therefore the larger of the two eigenvalues of $A^TA$ is $λ=\frac{3}{2}\pm\frac{1}{2}\sqrt{5}$, giving $||A||=\sqrt{\frac{3}{2}\pm\frac{1}{2}\sqrt{5}}$."

I don't get how it got that answer. I tried getting the eigenvalues of $A$, but it's $λ=1$.

Your calculation of the determinant is correct. To find the roots, just use the quadratic formula,
$$ax^2 + bx +c =0 \implies x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
Here,
$$\begin{align*}
&\lambda^2 - 3 \lambda + 1 = 0 \\
&\implies \lambda = \frac{-(-3) \pm \sqrt{(-3)^2 - 4(1)(1)}}{2(1)} = \frac{3 \pm \sqrt{9-4}}{2} =\frac{3 \pm \sqrt 5}{2}
\end{align*}$$
