Hiiiii Sakuraiiiii!



 sakuraiiiii:

I just want to do an optimization problem with constriants. With Lagrange Multiplier and KKT conditions, the problem could be converted to an unconstrained problem.


To expand a little on what Alban has said, the problem is as follows:
If I have an optimization problem where I seek to minimize a loss
function (e.g., a neural-network training loss), but then add constraints
to it, it is true that I can use Lagrange multipliers to convert the
constrained optimization to an unconstrained optimization.
However, this unconstrained optimization problem is no longer a
minimization problem, but rather one of finding a stationary point
(also called a critical point) of the augmented loss function.
That is, the gradient (including the gradient with respect to the
Lagrange multipliers) will vanish at the stationary-point solution, but
this solution need not be a minimum (with respect to the Lagrange
multipliers).
A simple example illustrates the difference:
We can minimize x**2 with gradient descent just fine, and the minimum
occurs ar x = 0 where the derivative vanishes.
The function x**3  also has a stationary point at x = 0 (where the
derivative vanishes), but gradient descent is no good for finding it.
Once x ends up less than zero – even by a little bit – gradient descent
will keep pushing x to become more and more negative, and off to
-inf.
I don’t know enough to state that you can’t use gradient descent with
Lagrange multipliers, but I don’t know how to do it.
If I were to try to hack something together for your problem, I would
try adding penalties for constraint violation to your loss function, and
use pytorch / gradient descent to minimize that.
So you could add, for example, alpha * h(x)**2 for your equality
constraints, and beta * min (g(x), 0)**2 for your inequality
constraints.  As you increase alpha and beta, minimizing your loss
function will increasing push you in the direction of satisfying your
constraints.  (Perhaps abs() would be better than **2, as it avoids
the “softer” quadratic minimum.)
You could say that adding these penalties to you loss function will
“encourage” your constraints to be close to being satisfied, but it won’t
force them to be satisfied.
Good luck.
K. Frank