How Does Quantization Affect Multilingual LLMs?
<|reference_start|>How Does Quantization Affect Multilingual LLMs?: Quantization techniques are widely used to improve inference speed and deployment of large language models. While a wide body of work examines the impact of quantized LLMs on English tasks, none have examined the effect of quantization across languages. We conduct a thorough analysis of quantized multilingual LLMs, focusing on their performance across languages and at varying scales. We use automatic benchmarks, LLM-as-a-Judge methods, and human evaluation, finding that (1) harmful effects of quantization are apparent in human evaluation, and automatic metrics severely underestimate the detriment: a 1.7% average drop in Japanese across automatic tasks corresponds to a 16.0% drop reported by human evaluators on realistic prompts; (2) languages are disparately affected by quantization, with non-Latin script languages impacted worst; and (3) challenging tasks such as mathematical reasoning degrade fastest. As the ability to serve low-compute models is critical for wide global adoption of NLP technologies, our results urge consideration of multilingual performance as a key evaluation criterion for efficient models.<|reference_end|>
@article{marchisio2024how,
    title={How Does Quantization Affect Multilingual LLMs?},
    author={Kelly Marchisio, Saurabh Dash, Hongyu Chen, Dennis Aumiller, Ahmet
  "Ust"un, Sara Hooker, Sebastian Ruder},
    journal={arXiv preprint arXiv:2407.03211},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03211},
    primaryClass={cs.CL cs.LG}
}