Stable and Unstable Manifolds Computation I want to compute (approximately) the stable and unstable manifolds around the origin of the following system.
$x' = 6x + 8y + (sinx)^2 - y^2$
$y' = 8x - 6y + x^2 + xy$
How can I solve it? Thank you very much.

Making $y=h(x) = \sum_{k=1}^n a_k x^k$ over the manifolds we have
$$
\dot y = h_x(x)\dot x
$$
or
$$
8x-6h(x)+x^2 + x h(x) = h_x(x)\left(6x+8h(x)+\left(x^2-\frac{x^4}{3}\right)-h^2(x)\right)
$$
considering $n = 3$ we have after reducing to a null polynomial
$$
\left\{
\begin{array}{l}
 4 \left(a_1+2\right) \left(2 a_1-1\right) =0\\
 -a_1^3+6 \left(4 a_1+3\right) a_2-1 =0\\
 16 a_2^2+\left(1-4 a_1^2\right) a_2+8 \left(4 a_1+3\right) a_3 =0\\
\end{array}
\right.
$$
and after solving we have
$$
\cases{
h_1(x) = -\frac{1183 x^3}{18000}+\frac{7 x^2}{30}-2 x\\
h_2(x) = -\frac{9 x^3}{16000}+\frac{3 x^2}{80}+\frac{x}{2}
}
$$
Those are two representations for unstable manifolds (in red) depicted in the following figure

NOTE
$\left(x^2-\frac{x^4}{3}\right)$ represents the first two term series representation for $(\sin x)^2$
