title: How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
summary: Meta's LLaMA family has become one of the most powerful open-source Large
  Language Model (LLM) series. Notably, LLaMA3 models have recently been released
  and achieve impressive performance across various with super-large scale
  pre-training on over 15T tokens of data. Given the wide application of low-bit
  quantization for LLMs in resource-limited scenarios, we explore LLaMA3's
  capabilities when quantized to low bit-width. This exploration holds the
  potential to unveil new insights and challenges for low-bit quantization of
  LLaMA3 and other forthcoming LLMs, especially in addressing performance
  degradation problems that suffer in LLM compression. Specifically, we evaluate
  the 10 existing post-training quantization and LoRA-finetuning methods of
  LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's
  low-bit quantization performance. Our experiment results indicate that LLaMA3
  still suffers non-negligent degradation in these scenarios, especially in
  ultra-low bit-width. This highlights the significant performance gap under low
  bit-width that needs to be bridged in future developments. We expect that this
  empirical study will prove valuable in advancing future models, pushing the
  LLMs to lower bit-width with higher accuracy for being practical. Our project
  is released on https://github.com/Macaronlin/LLaMA3-Quantization and quantized
  LLaMA3 models are released in https://huggingface.co/LLMQ.
paper_authors._id: 662712bd97eadbff167d0c78
paper_authors.name: Wei Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-23T07:15:11.751Z
paper_authors._id: 656db3f53dc1d277e5a64410
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/656db3f53dc1d277e5a64410/9kiY2K3MCRcBDk7MrkTBK.png
paper_authors.fullname: Wei Huang
paper_authors.type: user
paper_authors.user: AaronHuangWei
paper_authors._id: 662712bd97eadbff167d0c79
paper_authors.name: Xudong Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:59:31.991Z
paper_authors._id: 640df5f38512ec51d7f1f53b
paper_authors.avatarUrl: /avatars/f3d049b3a7db98437978f8646741ebe4.svg
paper_authors.fullname: Xudong Ma
paper_authors.type: user
paper_authors.user: Macaronlin
paper_authors._id: 662712bd97eadbff167d0c7a
paper_authors.name: Haotong Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:59:38.224Z
paper_authors._id: 65c49589c0b1921e19260a8d
paper_authors.avatarUrl: /avatars/7ce9af8c627f2a0c3db6bde82290ee1f.svg
paper_authors.fullname: Haotong Qin
paper_authors.type: user
paper_authors.user: HaotongQin
paper_authors._id: 662712bd97eadbff167d0c7b
paper_authors.name: Xingyu Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:59:44.716Z
paper_authors._id: 64612660933afb0106a9dee3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/Ea83e0zR_2m8foWy6J0AF.jpeg
paper_authors.fullname: Xingyu Zheng
paper_authors.type: user
paper_authors.user: Xingyu-Zheng
paper_authors._id: 662712bd97eadbff167d0c7c
paper_authors.name: Chengtao Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-23T07:59:57.501Z
paper_authors._id: 64e9bfc3f494f8b2a061a010
paper_authors.avatarUrl: /avatars/e55cfea55b45b03d1abfa38db6af58b6.svg
paper_authors.fullname: 吕呈滔
paper_authors.type: user
paper_authors.user: lvchengtao
paper_authors._id: 662712bd97eadbff167d0c7d
paper_authors.name: Hong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:52:03.496Z
paper_authors._id: 64f880358a4c5c401d881ac4
paper_authors.avatarUrl: /avatars/a50fff2386b4427a081ce26d9f877462.svg
paper_authors.fullname: ChenHong
paper_authors.type: user
paper_authors.user: chhhhhhh
paper_authors._id: 662712bd97eadbff167d0c7e
paper_authors.name: Jie Luo
paper_authors._id: 662712bd97eadbff167d0c7f
paper_authors.name: Xiaojuan Qi
paper_authors._id: 662712bd97eadbff167d0c80
paper_authors.name: Xianglong Liu
paper_authors._id: 662712bd97eadbff167d0c81
paper_authors.name: Michele Magno
paper_title: How Good Are Low-bit Quantized LLaMA3 Models? An Empirical Study
paper_summary: Meta's LLaMA family has become one of the most powerful open-source Large
  Language Model (LLM) series. Notably, LLaMA3 models have recently been released
  and achieve impressive performance across various with super-large scale
  pre-training on over 15T tokens of data. Given the wide application of low-bit
  quantization for LLMs in resource-limited scenarios, we explore LLaMA3's
  capabilities when quantized to low bit-width. This exploration holds the
  potential to unveil new insights and challenges for low-bit quantization of
  LLaMA3 and other forthcoming LLMs, especially in addressing performance
  degradation problems that suffer in LLM compression. Specifically, we evaluate
  the 10 existing post-training quantization and LoRA-finetuning methods of
  LLaMA3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMA3's
  low-bit quantization performance. Our experiment results indicate that LLaMA3
  still suffers non-negligent degradation in these scenarios, especially in
  ultra-low bit-width. This highlights the significant performance gap under low
  bit-width that needs to be bridged in future developments. We expect that this
  empirical study will prove valuable in advancing future models, pushing the
  LLMs to lower bit-width with higher accuracy for being practical. Our project
  is released on https://github.com/Macaronlin/LLaMA3-Quantization and quantized
  LLaMA3 models are released in https://huggingface.co/LLMQ.
paper_ai_summary: LLaMA3 performance degrades significantly when quantized to low bit-width, highlighting challenges in low-bit quantization for large language models.
paper_ai_keywords: Large Language Model LLaMA LLaMA3 low-bit quantization resource-limited post-training quantization LoRA-finetuning ultra-low bit-width performance degradation LLM compression