Is the matrix $A^4+A^3-3A^2-3A$ invertible? 
Let $A$ be a real $5 \times 5$ matrix satisfying $A^3-4A^2+5A-2I=O$. Is the matrix $A^4+A^3-3A^2-3A$ invertible?

Consider the polynomial $t^3-4t^2+5t-2=(t-1)^2(t-2)$. The minimal polynomial of $A$ divides $(t-1)^2(t-2)$. Let $p(t)$ be the minimal polynomial of $A$.
If $p(x)=t-1$ or $t-2$, then $A$ is a scalar multiple of the identity and $A^4+A^3-3A^2-3A=-4I \text{ or }6I$, so $A^4+A^3-3A^2-3A$ is invertible.
If $p(t)=(t-1)^2(t-2)$ or $(t-1)(t-2)$, then $A$ is similar to a upper triangular matrix $J$ with diagonal entries $1,2$. Then $A^4+A^3-3A^2-3A=P(J^4+J^3-3J^2-3J)P^{-1}$ and $\det(A^4+A^3-3A^2-3A)=\det(J^4+J^3-3J^2-3J)$. Since $J^4+J^3-3J^2-3J$ is upper triangular and the diagonal entries do not vanish, $\det(J^4+J^3-3J^2-3J) \ne 0$.
If $p(t)=(t-1)^2$, then by the same reasoning, $A^4+A^3-3A^2-3A$ is invertible.
Is there more efficient way to solve this kind of problems or I have to discuss any possible situations every time?

The long division of $x^4+x^3-3x^2-3x$ by $x^3-4x^2+5x-2$ leaves a remainder $12x^2-26x+10$, so your matrix to be checked for invertibility is
$$
2(6A^2-13A+5I)
$$
Suppose $v\ne0$ is in the null space of $6A^2-13A+5I$, so $6A^2v-13Av+5v=0$.
We also know that $A^3v-4A^2v+5Av-2v=0$. Multiplying the former by $A$, the latter by $6$ and subtracting, we obtain $11A^2v-25Av+12v=0$ and an elimination from the two “degree two” relations yields
$$
7Av-17v=0
$$
which means that $A$ should have $17/7$ as an eigenvalue. However, this is not possible, because an eigenvalue $\lambda$ of $A$ must satisfy $\lambda^3-4\lambda^2+5\lambda-2=0$.

The above doesn't use the Jordan canonical form. If you know it, you can prove that the eigenvalues of $p(A)$ (for $p$ a polynomial) have the form $p(\lambda)$ where $\lambda$ is an eigenvalue of $A$. Since the eigenvalues of $A$ must satisfy $(\lambda-1)^2(\lambda-2)=0$, they can only be $1$ or $2$; however
$$
x^4+x^3-3x^2-3x=x(x^2-3)(x+1)
$$
and neither $1$ nor $2$ are roots of this polynomial.
