A.2.2 TIMING EXPERIMENT SETUP
Our timing experiments are performed following the standard HuggingFace/accelerate4 setup also
used by the recent work LLM.int8() (Dettmers et al., 2022). In this setting, the model is split by
distributing chunks of consecutive layers across GPUs. Importantly, in this setup the communication
costs are minimal, < 5% of the total runtime even when working with 8 GPUs. This means almost
all of the reported speedups are due to our quantized-matrix full-precision vector product kernels.
We emphasize that the only difference between the FP16 baseline and our quantized models are the
kernels used to perform the underlying matrix-vector products.
This means all overheads due to HuggingFace, attention or non-quantized operations like residuals
or LayerNorms are exactly the same. Consequently, our quantized models should beneﬁt from more
advanced distribution strategies (Zheng et al., 2022) or more efﬁcient attention kernels (Dao et al.,
GPTQ