Find the polynomial $P(x)$ s.t. for any $a,b\in\Bbb C$ s.t. $a^2+b^2=ab$, $P(a+b)=6\big(P(a)+P(b)\big)+15a^2b^2(a+b)$. 
Suppose that $P(x)$ is a polynomial such that 
  $$P(a+b)=6\big(P(a)+P(b)\big)+15a^2b^2(a+b)$$
  for any complex numbers $a$ and $b$ satisfying $a^2+b^2=ab$, then find the polynomial $P(x)$.

I don't know how to even approach this one, it seems very complex. Your help is appreciated!

In this solution, $P$ is assumed to be an entire function (which is more generalised than just being a polynomial).  Let $\omega_1,\omega_2$ be roots of $x^2-x+1=0$. 
 We note that when $a=\omega_jt$ and $b=t$, we have $a^2+b^2=ab$, so that
$$P\big((\omega_j+1)t\big)-6\big(P(\omega_j t)-6P(t)\big)-15\omega_j^2(\omega_j+1)t^5=0\tag{1}$$
for $j=1,2$.  
Suppose that
$$P(x)=\sum_{k=0}^\infty p_kx^k.$$
If $k\neq 5$, then the coefficient of $t^k$ of the LHS of $(1)$ is $p_k\cdot f_k(\omega_j)$, where
$$f_k(x)=(x+1)^k-6(x^k+1).$$
If $p_k\ne 0$, then $f_k(\omega_j)=0$.  For $k\geq 5$, we have
$$\big|(\omega_j+1)^k\big|\geq |\omega_j+1|^k=\sqrt{3}^k=9\sqrt{3}$$
and
$$\big|6(\omega_j^k+1)\big|\leq 6\big(|\omega_j|^k+1\big)=6\cdot 2=12.$$
Thus
$$\big|(\omega_j+1)^k-6(\omega_j^k+1)\big|\geq \big|(\omega_j+1)^k\big|-\big|6(\omega_j^k+1)\big|\geq 9\sqrt{3}-12>0\,.$$
For $k=0,1,2,3,4$, we can easily verify by hand that
$$(\omega_j+1)^k-6(\omega_j^k+1)\ne 0.$$ 
For $k=0,1$, this is trivial.  For $k=2$, we note that $(x+1)^2-6(x^2+1)=-5x^2+2x-5$ is not divisible by $x^2-x+1$.  For $k=3$, we note that $\omega_j^3=-1$ and $(\omega_j+1)^3\ne 0$.  For $k=4$, note that $$(\omega_j+1)^4=\big((\omega_j+1)^2\big)^2=(3\omega_j)^2=9\omega_j^2$$ and $$\omega_j^4=-\omega_j,$$ so $$(\omega_j+1)^4-6(\omega_j^4+1)=9\omega_j^2+6\omega_j-6\ne0.$$
Thus $\omega_j$ is not a root of $f_k(x)$ for any integer $k\geq 0$.  This is a contradiction, so $p_k=0$ for $k\ne 5$.
The coefficient of $t^5$ in $(1)$ is
$$p_5(\omega_j+1)^5-6(\omega_j^5+1)-15\omega_j^2(\omega_j+1).\tag{2}$$
Noting that
$$(x+1)^5-6(x^5+1)-15x^2(x+1)=-5(x+1)(x^2-x+1)^5,$$
we conclude that
$$(\omega_j+1)^5-6(\omega_j^5+1)-15\omega_j^2(\omega_j+1)=0.\tag{3}$$
Subtract $(3)$ from $(2)$ to get
$$(p_5-1)(\omega_j+1)^5=0.$$
Therefore $p_5=1$, so $P(x)=x^5$.
