Question: The system in question is:
$$
x' = y\\
y' = \mu x+x^2
$$
This has a fixed point at $x=0,-\mu$ and $y=0$ and after computing the Jacobian I find eigenvalues of
$$
\lambda = \pm \sqrt{\mu}
$$
for the $x=0 ,y=0$ solution. This is a saddle point for $\mu>0$ but becomes completely imaginary for $\mu<0.$ I am not sure how to determine the stability for $\mu<0$

Answer: 
You cannot conclude anything about the stability of a fixed point through linearization whenever the Jacobian has at least one purely imaginary eigenvalue.

However, what you can do, is to express the system as a second-order system by letting $\ddot{x} = \dot{y}$:

$$\ddot{x}-\mu x - x^2 = 0$$

We can interpret this system as a generalized harmonic oscillator with an amended potential. To see this, recall that the equation for a harmonic oscillator is $\ddot{x}+x=0$, where the spring constant is 1 and the potential is given by $\int\_0^x y \ dy = x^2/2$. So in our case, we have a negative spring constant when $\mu>0$ and a negative spring constant when $\mu<0$. The potential in our case is given by:

$$\int\_0^x (-\mu y -y^2) \ dy = -\mu x^2/2 - x^3/3$$

Furthermore, you can have more fun by writing down the Hamiltonian for your system:

$$H = T + V = \dot{x}^2/2 - \mu x - x^2 = p^2/2 - \mu x - x^2$$

So now you can draw the potential and use it to sketch the phase portrait of your system, from which you can infer the stability of the equilibrium points.

Here is an example of what this looks like for a different system with a different potential:

[![Using the potential to sketch the phase portrait](https://i.stack.imgur.com/1czIw.png)](https://i.stack.imgur.com/1czIw.png)

So basically, for your system, you can look at the potential and say that the quadratic term dominates over the cubic when $x$ is close to 0, which means that for $\mu<0$ the potential will have a valley at $x=0$ and, therefore, solutions that start around it on a sufficiently low energy level, will get trapped (i.e., the equilibrium point $x=0,y=0$ is a center).

Hope this helps!
Note that this system has the symmetry ($x \to x, y \to -y, t \to -t$). Thus a trajectory that starts out on the $x$ axis and returns to the $x$ axis is reflected across that axis to make a closed loop.
The fixed point is a centre: stable but not asymptotically stable.