Our work introduces a general method for compressing large language models (LLMs) via quan-
tization, with little-to-no accuracy loss in terms of standard accuracy metrics such as perplexity.
Our method is task-agnostic, as it only uses a tiny amount of randomly-chosen data for calibration.
We therefore do not foresee any signiﬁcant ethical implications arising directly from the technical
details of our method. However, one possible consideration is that our study focused on “leading
accuracy” metrics that are standard in the literature, such as perplexity, which is essentially standard
in the literature (Dettmers et al., 2022; Yao et al., 2022). We believe a thorough study of the impact
of compression upon secondary measures, and in particular bias effects (Bender et al., 2021) is war-
ranted, and may be rendered easier through our work. At the same time, our work makes inference
on extremely large language models more accessible, for better or for worse. We believe that, in
GPTQ