Problem 74. Consider the positive real constants $m, n$, such that $3 n^{2}>m^{2}$. For real numbers $a, b, c$ such that $a+b+c=m, a^{2}+b^{2}+c^{2}=n^{2}$, find the maximum and minimum of
$$P=a^{2} b+b^{2} c+c^{2} a$$
SOLUTION. Let \(a=x+\frac{m}{3}, b=y+\frac{m}{3}, c=z+\frac{m}{3}\). From the given conditions, we get that \(x+y+z=0\) and \(x^{2}+y^{2}+z^{2}=\frac{3 n^{2}-m^{2}}{3}\). The expression \(P\) becomes
\[P=x^{2} y+y^{2} z+z^{2} x+\frac{m^{3}}{9}\]

Notice that
\[\begin{array}{l}
\sum_{c y c}\left(3 x \sqrt{\frac{2}{3 n^{2}-m^{2}}}-\frac{18 x y}{3 n^{2}-m^{2}}-1\right)^{2} \\
=3+\frac{18}{3 n^{2}-m^{2}}\left(\sum_{c y c} x\right)^{2}+\frac{324}{\left(3 n^{2}-m^{2}\right)^{2}} \sum_{c y c} x^{2} y^{2} \\
-6 \sqrt{\frac{2}{3 n^{2}-m^{2}}} \sum_{c y c} x-54\left(\frac{2}{3 n^{2}-m^{2}}\right)^{3 / 2} \sum_{c y c} x^{2} y \\
=3+\frac{324}{\left(3 n^{2}-m^{2}\right)^{2}} \sum_{c y c} x^{2} y^{2}-54\left(\frac{2}{3 n^{2}-m^{2}}\right)^{3 / 2} \sum_{c y c} x^{2} y
\end{array}\]

Since \(x+y+z=0\), we get \(x y+y z+z x=-\frac{1}{2}\left(x^{2}+y^{2}+z^{2}\right)=-\frac{3 n^{2}-m^{2}}{6}\). Therefore
\[\sum_{c y c} x^{2} y^{2}=\left(\sum_{c y c} x y\right)^{2}-2 x y z \sum_{c y c} x=\left(\sum_{c y c} x y\right)^{2}=\frac{\left(3 n^{2}-m^{2}\right)^{2}}{36}\]
and we get
\[12-54\left(\frac{2}{3 n^{2}-m^{2}}\right)^{3 / 2} \sum_{c y c} x^{2} y \geq 0\]
or in other words,
\[\sum_{c y c} x^{2} y \leq \frac{2}{9}\left(\frac{3 n^{2}-m^{2}}{2}\right)^{3 / 2}\]

If we choose
\[x=\frac{\sqrt{2\left(3 n^{2}-m^{2}\right)}}{3} \cos \frac{2 \pi}{9}, y=\frac{\sqrt{2\left(3 n^{2}-m^{2}\right)}}{3} \cos \frac{4 \pi}{9}, z=\frac{\sqrt{2\left(3 n^{2}-m^{2}\right)}}{3} \cos \frac{8 \pi}{9}\]
then
\[P=\frac{2}{9}\left(\frac{3 n^{2}-m^{2}}{2}\right)^{3 / 2}+\frac{m^{3}}{9}\]

So
\[\max P=\frac{2}{9}\left(\frac{3 n^{2}-m^{2}}{2}\right)^{3 / 2}+\frac{m^{3}}{9}\]

Similarly, by considering the expression
\[\sum_{c y c}\left(3 x \sqrt{\frac{2}{3 n^{2}-m^{2}}}+\frac{18 x y}{3 n^{2}-m^{2}}+1\right)^{2}\]
we easily conclude that
\[\min P=-\frac{2}{9}\left(\frac{3 n^{2}-m^{2}}{2}\right)^{3 / 2}-\frac{m^{3}}{9}\]

The problem is completely solved.