Find parameters $a,b$ such that $x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1-\left(x^3-x^2+a x+b\right)^2>0$ The probrem is to prove that
$$x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1>0.$$
(the minimum value is about 0.02, tested by wolframalpha.)
I use sos(sum of squares) method, my idea is to reduce the degree of the polynomial gradually.
First I need to find $a,b$ so that
$$x^6-2 x^5+2 x^4+2 x^3-x^2-2 x+1-\left(x^3-x^2+a x+b\right)^2>0.$$
It's equivalent to
$$(1-2 a) x^4+(2 a-2 b+2)x^3 +\left(-a^2+2 b-1\right)x^2 +(-2 a b-2)x -b^2+1>0,$$ or
$$
\left( (1-2\!\:a)\!\:x^4+(2+2\!\:a-2\!\:b)\!\:x^3+(-1-a^2+2\!\:b)\!\:x^2+(-2-2\!\:a\!\:b)\!\:x+1-b^2 \right) _{\min}>0.$$
Let
$$f(x)=
(1-2\!\:a)\!\:x^4+(2+2\!\:a-2\!\:b)\!\:x^3+(-1-a^2+2\!\:b)\!\:x^2+(-2-2\!\:a\!\:b)\!\:x+1-b^2$$
and $f'(x)=0$, I get
$$
-2-2\!\:a\!\:b+2\!\:(-1-a^2+2\!\:b)\!\:x+3\!\:(2+2\!\:a-2\!\:b)\!\:x^2+4\!\:(1-2\!\:a)\!\:x^3=0.
$$ However, it's difficult  and ugly to solve the equation. And it's impossible to solve the extreme point when the degree of the polynomial increases, for example, prove that $$x^{12}-x^{11}+x^8-x^7+x^6+x^3-2 x+1>0,$$ we need to find 5 parameters $a,b,c,d,e$ such that $$
x^{12}-x^{11}+x^8-x^7+x^6+x^3-2x+1-\left( x^6-\frac{x^5}{2}+ax^4+bx^3+cx^2+dx+e \right) ^2>0.
$$
How can I solve the problem? Is there any general method?

For the original problem, observe that
$$x^6 - 2x^5 + 2x^4 + 2x^3 - x^2 - 2x + 1 = (x^3 - x^2)^ 2 + (x^2 +x - 1)^2.$$
Since both squares cannot be 0 at the same time, hence it is strictly positive.

For your stated problem, $ a, b = 0 $ is still not a solution to your strict inequality because the expression does hit 0 when $x^2 + x - 1 = 0 $.
Testing values near that point indicates that $ a = -0.01, b = 0.01$ works.
The resulting quartic has 4 complex roots, hence is strictly positive.
Proof by Wolfram Alpha since it's not that interesting.
Given the above, I'm not motivated to figure out the rest of the SOS.

Notes

*

*You do not always have to "reduce the degree of the polynomial gradually".

*

*EG We could have (say) $ f(x) = (0.6x^3 + g(x) ) ^2 + (0.8 x^3 + h(x) ) ^2 $ for some quadratics $g(x), h(x)$.

*EG Had you tried to do so, you'd likely have ended up with $ (x^3 - x^2 + 0.5 x + 1.5)^2$ in order to remove the $x^4$ and then $x^3 $ term. However, this leaves us with $1.75x^2 - 3.5x - 1.25$ which has distinct real roots, so we can't SOS further.



*I chanced upon this identity, in part from recognizing some of the terms. You guessed that $(x^3-x^2)^2$ might be involved, so looking at $ f(x) - (x^3 -x^2)^2$  would have been a good next step.

*The minimum of $f(x)$ occurs near the positive root of $x^2 + x - 1 = 0$. This shouldn't be too surprising based on the identity.

