a few billion parameters. The main issue appears to be the repeated applications of Equation (5),
which accumulate various numerical errors, especially through the additional matrix inversion.
For smaller models, applying dampening, that is adding a small constant λ (we always choose 1% of
the average diagonal value) to the diagonal elements of H appears to be sufﬁcient to avoid numerical
issues. However, larger models require a more robust and general approach.
To address this, we begin by noting that the only information required from H−1
, where Fq denotes
the set of unquantized weights when quantizing weight q, is row q, or more precisely, the elements in
this row starting with the diagonal. The consequence is that we could precompute all of these rows
using a more numerically-stable method without any signiﬁcant increase in memory consumption.
Indeed, the row removal via (3) for our symmetric H−1 essentially corresponds to taking a Cholesky
GPTQ