Finding maximum value of $x^2y$ given following condition. 
Find the maximum value of $x^2y$ given the constraint that : $$x+y+\sqrt{2x^2+2xy+3y^2} =k ~\text{(constant)}~ \text{and} ~~ x,y \in \mathbb{R^+}$$ (Answer is in terms of '$k$')

I tried using AM-GM , $$\frac x2+\frac x2+y \ge 3\Bigg(\frac{x^2y}{4}\Bigg)^{\frac13}$$
And the same thing : $$\frac{2x^2}{3}+\frac{2x^2}{3}+\frac{2x^2}{3}+xy+xy+ 3y^2 \ge 6\Bigg(\frac{8}{3} (x^2y)^4 \Bigg)^{\frac 16}$$
But when I solved for upper bound of $x^2y$ using these two inequalities, my answer didn't match. Where am I wrong? Can you please help by giving a correct way of solving this question?
Thanks.

I think you mean that the variables are positives.
By the given we obtain:
$$2x^2+2xy+3y^2=(k-x-y)^2$$ or
$$x^2+2y^2+2k(x+y)=k^2$$ or
$$(x+k)^2+2\left(y+\frac{k}{2}\right)^2=\frac{5k^2}{2}$$ or
$$2(x+k)^2+(2y+k)^2=5k^2.$$
Now, by AM-GM and Holder we obtain:
$$5k^2=2(x+k)^2+(2y+k)^2\geq3\sqrt[3]{\left((x+k)^2(2y+k)\right)^2}\geq$$
$$\geq3\sqrt[3]{\left(\left(\sqrt[3]{2x^2y}+k\right)^3\right)^2}=3\left(\sqrt[3]{2x^2y}+k\right)^2.$$
Thus, $$\sqrt[3]{2x^2y}+k\leq k\sqrt{\frac{5}{3}}$$ or
$$x^2y\leq\frac{k^3}{2}\left(\sqrt{\frac{5}{3}}-1\right)^3.$$
The equality occurs for $x=2y$, which gives the answer:
$$\frac{k^3}{2}\left(\sqrt{\frac{5}{3}}-1\right)^3.$$
Done!
