Question: Say we had $$y'' +2y' -8y = xe^{-x}.$$ For the particular solution, would it be wise to try $$y\_p(x) = (c\_1 + c\_2x)e^{-x}\quad or\quad y\_p(x) = c\_1xe^{-x}?$$ Similarly, we had $y'' +2y' -8y = x^2e^{-x}$ or $ y'' +2y' -8y = x^2cos(x)$ would it be wise to try $y\_p(x) = (c\_1 + c\_2x + c\_3x^2)e^{-x} $ and $y\_p(x) = (c\_1 + c\_2x + c\_2x^2)(\sin(x) + \cos(x))$?

Answer: 
In general, if you have an equation of the form
$$
y^{(n)}+a\_{n-1}y^{(n-1)}+\cdots a\_0 y^{(0)}=p(x)\,\mathrm{e}^{ax},
$$
where $p$ is a polynomial of degreee $k$, then you look for a solution of the form $q(x)\,\mathrm{e}^{ax}$, where $q$ is also a polynomial of degree $k$, unless $a$ is a root of the characteristic polynomial. In the latter case you look for a solution of the form $q(x)\,\mathrm{e}^{ax}$, where $q$ is a polynomial of degree $k+m$, where $m$ is the multiplicity of the root $a$ in the characteristic polynomial.

In your case, you look for a special solution of the form $y=(A+Bx)\mathrm{e}^{-x}$, but the general solution is of the form:
$$
y=(A+Bx)\mathrm{e}^{-x}+c\_1\mathrm{e}^{2x}+c\_1\mathrm{e}^{-4x}
$$
For $x^2\cos x$ you should look for a special solution of the form
$$
p\_1(x)\cos x+p\_2(x)\sin x,
$$
where $p\_1$, $p\_2$ are polynomials of degree at most 2.
A good criteria:

> 
> Let $a\_ny^{(n)}+a\_{n-1}y^{(n-1)}+\cdots+a\_1y'+a\_0y=Q(x)$ where $a\_n\neq0$ and $Q(x)$ is not zero in an interval $I$. If *No term* of $Q(x)$ is the same as a term of $y\_c(x)$, then the particular solution $y\_p(x)$ is a linear combination of the terms in $Q(x)$ and all its linearly independent derivatives.
> 
> 
> 

Here, we have $y\_c(x)=C\_1e^{2x}+C\_2e^{-4x}$ so according to what above point says we get: $$y\_p(x)=(Ae^x+Bxe^x)$$ Now we can use the [Undetermined coefficient method](http://en.wikipedia.org/wiki/Method_of_undetermined_coefficients) to find $A$ and $B$.