When is matrix $A$ diagonalizable? I got the following matrix:
$$ A = 
        \begin{pmatrix}
        a & 0 & 0 \\
        b & 0 & 0 \\
        1 & 2 & 1 \\
        \end{pmatrix}
$$
I need to answer when this matrix is diagonalizable.
Its characteristic polynomial is $ t(t-a)(t-1) $. So its 3 eigenvalues are 0, 1 and a. Both the algebraic and geometry multiplicities of those values are 1 (for all of them).
Let's look at the matrices for those eigenvalues:
$$ A - 0I = 
        \begin{pmatrix}
        a & 0 & 0 \\
        b & 0 & 0 \\
        1 & 2 & 1 \\
        \end{pmatrix}
$$
$$ A - I = 
        \begin{pmatrix}
        a -1 & 0 & 0 \\
        b & -1 & 0 \\
        1 & 2 & 0 \\
        \end{pmatrix}
$$
$$ A  - aI = 
        \begin{pmatrix}
        0 & 0 & 0 \\
        b & -a & 0 \\
        1 & 2 & 1 - a \\
        \end{pmatrix}
$$
$\rho (A - 0I) = 2 $
$\rho (A - 1I) = 2 $
$\rho (A - aI) = 2 $
It seems that for every $a$ and $b$ this matrix would be diagonalizable.
But it's not. Where am I wrong?

Your matrix has at least $2$ eigenvalues, namely $0$ and $1$, and maybe a third, namely$~a$ it it is different from those two others. In the latter case we have $3$ simple roots of the characteristic polynomial, and $A$ is automatically diagonalisable. So the remaining interesting case is $a\in\{0,1\}$.
In that case $A$ is diagonalisable if and only if the polynomial $(X-0)(X-1)=X^2-X$ annihilates $A$, in other words if $A^2-A=0$ (the kernel of $A^2-A$ is the sum of the eigenspaces of $\lambda=0$ and $\lambda=1$). Now 
$$
A^2-A = 
        \begin{pmatrix}
        a^2-a & 0 & 0 \\
        b(a-1) & 0 & 0 \\
        a+2b & 0 & 0 \\
        \end{pmatrix}
$$
so in the remaining cases $A$ is diagonalisable only if either $a=0$ and $b=0$, or $a=1$ and $b=-\frac12$.
