8

Published as a conference paper at ICLR 2023

Figure 3: The accuracy of OPT and BLOOM models post-GPTQ, measured on LAMBADA.

Additional Tricks. While our experiments so far have focused exclusively on vanilla row-wise
quantization, we want to emphasize that GPTQ is compatible with essentially any choice of quanti-
zation grid. For example, it is easily combined with standard grouping (Alistarh et al., 2017; Park
et al., 2022), i.e. applying independent quantization to groups of g consecutive weights. As shown in
the last rows of Table 5, this can bring noticeable extra accuracy for the largest models at 3-bit. Fur-
ther, as visualized in Figure 4, it signiﬁcantly reduces the accuracy losses for medium sized models
at 4-bit precision.

Model
OPT-175B
BLOOM

FP16
8.34
8.11

g128
9.58
9.55

g64
9.18
9.17

g32
8.94
8.83

3-bit
8.68
8.64

Table 7: 2-bit GPTQ quantization results with
varying group-sizes; perplexity on WikiText2.
GPTQ