  Satisfaction of the strict saddle property has become a standard assumption
in non-convex optimization, and it ensures that many first-order optimization
algorithms will almost always escape saddle points. However, functions exist in
machine learning that do not satisfy this property, such as the loss function
of a neural network with at least two hidden layers. First-order methods such
as gradient descent may converge to non-strict saddle points of such functions,
and there do not currently exist any first-order methods that reliably escape
non-strict saddle points. To address this need, we demonstrate that
regularizing a function with a linear term enforces the strict saddle property,
and we provide justification for only regularizing locally, i.e., when the norm
of the gradient falls below a certain threshold. We analyze bifurcations that
may result from this form of regularization, and then we provide a selection
rule for regularizers that depends only on the gradient of an objective
function. This rule is shown to guarantee that gradient descent will escape the
neighborhoods around a broad class of non-strict saddle points, and this
behavior is demonstrated on numerical examples of non-strict saddle points
common in the optimization literature.

Optimization and Control