Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs
<|reference_start|>Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs: We introduce Integer Scale, a novel post-training quantization scheme for large language models that effectively resolves the inference bottleneck in current fine-grained quantization approaches while maintaining similar accuracies. Integer Scale is a free lunch as it requires no extra calibration or fine-tuning which will otherwise incur additional costs. It can be used plug-and-play for most fine-grained quantization methods. Its integration results in at most 1.85x end-to-end speed boost over the original counterpart with comparable accuracy. Additionally, due to the orchestration of the proposed Integer Scale and fine-grained quantization, we resolved the quantization difficulty for Mixtral-8x7B and LLaMA-3 models with negligible performance degradation, and it comes with an end-to-end speed boost of 2.13x, and 2.31x compared with their FP16 versions respectively.<|reference_end|>
@article{li2024integer,
    title={Integer Scale: A Free Lunch for Faster Fine-grained Quantization of LLMs},
    author={Qingyuan Li, Ran Meng, Yiduo Li, Bo Zhang, Yifan Lu, Yerui Sun, Lin
  Ma, Yuchen Xie},
    journal={arXiv preprint arXiv:2405.14597},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.14597},
    primaryClass={cs.LG cs.AI}
}