Hi Lalla!



 lallazhao:

Apparently, some negative value exists in the eigenvalues. However, the Hessian matrix of logistic regression is supposed to be positive semi-definite


This is a numerical issue.  (Your code for calculating the Hessian looks
right.)
What’s at issue is that you are working with a 100 x 100 matrix (where
the 100 in question is the input_dim of your Linear, not the number of
samples in your batch).
As a matrix gets larger, it is common for it to become more ill-conditioned.
The so-called condition number of a matrix tells you, roughly, how much
any error (including numerical round-off error) might get “amplified” when
doing things like inverting the matrix or calculating its eigenvalues.  One
common definition of the condition number is the ratio of the magnitudes
of the largest and smallest (in magnitude) eigenvalues of the matrix.
You can use torch.linalg.cond() to calculate the condition number of your
Hessian matrix.
You can do two things to explore this:
First, look at different size matrices.  Repeat the calculation with increasing
values for input_dim, say, 5, 10, 20, 50, 100, and see how the condition
number grows as input_dim grows.  (You may keep using 100 for the
batch size.)  Single-precision floating-point arithmetic is accurate to
about 7 decimal digits.  When your condition number grows to the order
of about 10^7, to can’t expect the result for your eigenvalues – especially
for the smaller eigenvalues – to remain accurate, and what should have
been a positive eigenvalue can become, numerically, negative.
You can also repeat the calculation in double precision (using the same
input data converted to double precision).  Treat the double precision as
the “correct” answer.  (The double-precision calculation will have the
same  numerical issues, but they won’t set in as soon.)  As long as the
single-precision and double-precision results agree reasonably well, they
should be reasonably accurate.  You should find that your eigenvalues
don’t become negative until the single-precision and double-precision
results start to deviate.
As a final comment, there are two ways numerically negative eigenvalues
could show up.  First, the Hessian matrix you calculate has numerical error
in it.  It could be that the Hessian matrix – with its numerical error – has
a negative eigenvalue.  Second, it could be that the Hessian matrix – even
with its numerical error – doesn’t have a negative eigenvalue, but that
numerical error in the eigenvalue computation produces a negative value.
I’m not sure which is more likely in this specific case.  Related to this, note
that your Hessian matrix, as calculated, will probably not be exactly
symmetric (due to round-off error).
Best.
K. Frank