evaluation details can be found in Appendix A.2.1.
To ensure that the entire compression procedure can be performed with signiﬁcantly less GPU mem-
ory than what would be required to run the full precision model, some care must be taken. Specif-
ically, we always load one Transformer block, consisting of 6 layers, at a time into GPU memory
and then accumulate the layer-Hessians and perform quantization. Finally, the current block inputs
are sent through the fully quantized block again to produce the new inputs for the quantization of
the next block. Hence, the quantization process operates not on the layer inputs in the full precision
model but on the actual layer inputs in the already partially quantized one. We ﬁnd that this brings
noticeable improvements at negligible extra cost.
Baselines. Our primary baseline, denoted by RTN, consists of rounding all weights to the nearest
quantized value on exactly the same asymmetric per-row grid that is also used for GPTQ, meaning
GPTQ