Solve $y^{(4)}-2y^{(3)}+2y'-y=xe^x.$ Solve $y^{(4)}-2y^{(3)}+2y'-y=xe^x.$
The characteristic equation is $(r-1)^3(r+1)\Rightarrow y_h=(C_1+C_2x+C_3x^2)e^x+C_4e^{-x}.$
The problem is the particular equation. Why doesn't it work with the ansatz
$$y=(ax+b)e^x?$$
I get 
\begin{array}{lcl}
y        & = & e^x(ax+b) \\
y'       & = & e^x(a(x+1)+b)\\
y''      & = & e^x(a(x+2)+b)  \\
y^{(3)}  & = & e^x(a(x+3)+b)  \\
y^{(4)}  & = & e^x(a(x+4)+b)
\end{array}
Setting these in i get $$e^x[((a(x+4)+b))-2((a(x+3)+b))+2(e^x(a(x+1)+b))-(e^x(ax+b))] = e^x\cdot 0=0.$$
So this doesn't work. Is it because I already have corresponding powers of $x$
in the homogenous solution? How can i fix my ansatz?

Solve step by step the equation $$(D-1)^3(D+1)y=xe^x$$ (where $D$ is differentiation operator) and there is no need to worry about guessing particular solution. Let $z=(D-1)^3y$ so that $$(D+1)z=xe^x$$ or $$D(e^xz) =xe^{2x}$$ or $$ze^x=\frac{xe^{2x}}{2}-\frac{e^{2x}}{4}+a$$ or $$z=\frac{xe^x} {2}-\frac{e^x}{4}+ae^{-x}$$ Next put $t=(D-1)^2y$ so that $$(D-1)t=z$$ or $$D(te^{-x})= ze^{-x} =\frac{x} {2}-\frac{1}{4}+ae^{-2x}$$ or $$t=e^x\left(\frac{x^2}{4}-\frac{x}{4}\right) -\frac{ae^{-x}}{2}+be^x$$ Since $a$ is arbitrary constant one can replace $-a/2$ by $a$ to get $$t=e^x\left(\frac{x^2}{4}-\frac{x}{4}\right)+ae^{-x}+be^x$$ Going further let $u=(D-1)y$ so that $$(D-1)u=t$$ or $$D(ue^{-x}) =te^{-x} =\frac{x^2}{4}-\frac{x}{4}+ae^{-2x}+b$$ or $$u=e^{x} \left(\frac{x^3}{12}-\frac{x^2}{8}\right)-\frac{ae^{-x}}{2}+bxe^x+ce^x$$ Replacing $-a/2$ by $a$ we get $$u=e^x\left(\frac{x^3}{12}-\frac{x^2}{8}\right)+ae^{-x}+bxe^x+ce^x$$ and in similar manner solving the final equation $(D-1)y=u$ we get $$y=e^x\left(\frac{x^4}{48}-\frac{x^3}{24}\right)+ae^{-x}+bx^2e^x+cxe^x+de^x$$ which is the desired solution. 
