Question: I have read a lot of articles on Big O notation but never found a problem like this and I have no idea how to solve it.

$$3 \cdot O(n^3) = O(3 + n^3)$$

Answer: 
(1)-(2) gives $a=b$ or $3(a+b)=-1$.

If $a=b$ then the first equation gives $a=b=0$ or $a=b=\frac{1}{2}$.

(1) gives $3(a+b)(a^2-ab+b^2)=3ab$. Substituting $3(a+b)=-1$ gives $(a+b)^2=0$, so $a+b=0$. Contradicting $3(a+b)=-1$.

So the only solutions are $(a,b)=(0,0)$ or $(\frac{1}{2},\frac{1}{2})$
From equation (2) you get that $(a-b)(3(a+b)+1)=0$. Then, either $a=b$ or $3(a+b)+1=0$.

In the first case we get, putting $b=a$ in equation (1), that $a^2(2a-1)=0$. So, either $a=b=0$, or $a=b=1/2$.

Assume now that $a\neq b$. Then $b=-1/3-a$. Putting this in equation (1) you get that

$$a^3-a(-1/3-a)+(-1/3-a)^3=0$$

this s equivalent to $-\frac{1}{27}=0$, which has no solutions.

Now you can take these possible solutions and check if they satisfy the original system of equations.