require careful adjustments for optimal performance on non-vision models. Overall, GPTQ appears
to be competitive with state-of-the-art post-training methods for smaller models, while taking only
< 1 minute rather than ≈ 1 hour. This enables scaling to much larger models.
Runtime. Next we measure the full model quantization time (on a single NVIDIA A100 GPU) via
GPTQ; the results are shown in Table 2. As can be seen, GPTQ quantizes 1-3 billion parameter
models in a matter of minutes and 175B ones in a few hours. For reference, the straight-through
based method ZeroQuant-LKD (Yao et al., 2022) reports a 3 hour runtime (on the same hardware)
for a 1.3B model, which would linearly extrapolate to several hundred hours (a few weeks) for 175B
GPTQ