Hi Thomas!



 tom:

Note that here, we know w.r.t. which variables the Lagrangian would be maximized and w.r.t. which ones it would be minimized, so in this way the problem is much easier than “find a saddle point”.


I don’t think it is as simple as this, the reason being that at the
constrained minimum, the gradient of the loss function plus the
Lagrange-multiplier terms in the direction of the Lagrange
multiplier is flat.
Consider the trivial example:
Minimize x**2 / 2, where x = 1.  To do so, we find the critical
point of x**2 / 2 + lambda * (x - 1).  The critical point is
(1, -1), and at the critical point the Hessian matrix is
[[1, 1], [1, 0]].  (The second partial with respect to lambda
is indeed 0.)
Letting phi = (1 + sqrt (5)) / 2 (the so-called “golden
ratio”), the Hessian’s eigenvectors (“principal axes”) are
(phi, 1) and  (1, -phi) (positive and negative curvature,
receptively).  These don’t align with the original variable, x, and
the Lagrange multiplier.
Even near the constrained minimum, I don’t see how you will
know in which directions to take “descent” steps and “ascent”
steps without calculating the second partials and looking at
the Hessian matrix.
Best.
K. Frank