Hi Lalla!



 lallazhao:

I want to confirm that, does turning the model into double is the only way to increase the numerical stability?


To first approximation, yes, using double precision is your best bet for
reducing this issue.
Please note, double precision won’t eliminate that problem – it will just
postpone it some, and perhaps not by very much.
One thing you could try that might help marginally:  Symmetrize your
Hessian matrix.  Due to round-off error, your version of hess is not
exactly symmetric.
Setting hess = (hess + hess.T) / 2 will force hess to be symmetric
and might make the elements of hess a little more accurate.  This won’t
solve your problem, but it might reduce is a little bit.
Also, what are you using hess for?  If it’s critical for hess to be
positive-definite, but it’s okay for hess (and, in particular, its smallest
eigenvalues) to not be fully correct, you could add a small epsilon
to it’s diagonal.
Another approach – depending upon how you use hess – would be
to only work in the subspace of hess for which the eigenvalues are
comfortably positive.  You can do this by setting any eigenvalues values
that lie below a certain threshold to zero, or, similarly, perform a
singular-value decomposition, and set any singular values below a
certain threshold to zero.
Best.
K. Frank