title: A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language
    Models: An Experimental Analysis up to 405B
summary: Prior research works have evaluated quantized LLMs using limited metrics such
  as perplexity or a few basic knowledge tasks and old datasets. Additionally,
  recent large-scale models such as Llama 3.1 with up to 405B have not been
  thoroughly examined. This paper evaluates the performance of instruction-tuned
  LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on
  models ranging from 7B to 405B. Using 13 benchmarks, we assess performance
  across six task types: commonsense Q\&A, knowledge and language understanding,
  instruction following, hallucination detection, mathematics, and dialogue. Our
  key findings reveal that (1) quantizing a larger LLM to a similar size as a
  smaller FP16 LLM generally performs better across most benchmarks, except for
  hallucination detection and instruction following; (2) performance varies
  significantly with different quantization methods, model size, and bit-width,
  with weight-only methods often yielding better results in larger models; (3)
  task difficulty does not significantly impact accuracy degradation due to
  quantization; and (4) the MT-Bench evaluation method has limited discriminatory
  power among recent high-performing LLMs.
paper_authors._id: 66ea30ceaeac53376f3b11d1
paper_authors.name: Jemin Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-18T07:01:44.887Z
paper_authors._id: 65b9dee19c4955ae7aee4954
paper_authors.avatarUrl: /avatars/263f129605c7763185c49076174b891b.svg
paper_authors.fullname: Jemin Lee
paper_authors.type: user
paper_authors.user: leejaymin
paper_authors._id: 66ea30ceaeac53376f3b11d2
paper_authors.name: Sihyeong Park
paper_authors._id: 66ea30ceaeac53376f3b11d3
paper_authors.name: Jinse Kwon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:08:41.717Z
paper_authors._id: 65d6b1d0dee9f442f8ebf205
paper_authors.avatarUrl: /avatars/9792889183b8e2718752ebed4fd1fcac.svg
paper_authors.fullname: Jinse Kwon
paper_authors.type: user
paper_authors.user: kwonse
paper_authors._id: 66ea30ceaeac53376f3b11d4
paper_authors.name: Jihun Oh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:09:04.894Z
paper_authors._id: 63620c69da0599dc08cb200b
paper_authors.avatarUrl: /avatars/9a6a035684e3765efd24517b81e241c5.svg
paper_authors.fullname: jihun oh
paper_authors.type: user
paper_authors.user: oj9040
paper_authors._id: 66ea30ceaeac53376f3b11d5
paper_authors.name: Yongin Kwon
paper_title: A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language
    Models: An Experimental Analysis up to 405B
paper_summary: Prior research works have evaluated quantized LLMs using limited metrics such
  as perplexity or a few basic knowledge tasks and old datasets. Additionally,
  recent large-scale models such as Llama 3.1 with up to 405B have not been
  thoroughly examined. This paper evaluates the performance of instruction-tuned
  LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on
  models ranging from 7B to 405B. Using 13 benchmarks, we assess performance
  across six task types: commonsense Q\&A, knowledge and language understanding,
  instruction following, hallucination detection, mathematics, and dialogue. Our
  key findings reveal that (1) quantizing a larger LLM to a similar size as a
  smaller FP16 LLM generally performs better across most benchmarks, except for
  hallucination detection and instruction following; (2) performance varies
  significantly with different quantization methods, model size, and bit-width,
  with weight-only methods often yielding better results in larger models; (3)
  task difficulty does not significantly impact accuracy degradation due to
  quantization; and (4) the MT-Bench evaluation method has limited discriminatory
  power among recent high-performing LLMs.
paper_ai_summary: Various quantization methods are evaluated on large instruction-tuned LLMs across multiple benchmarks, revealing that quantized larger models often perform better than smaller FP16 models, except in some specific tasks.
paper_ai_keywords: quantized LLMs GPTQ AWQ SmoothQuant FP8 instruction-tuned LLMs commonsense Q&A knowledge and language understanding instruction following hallucination detection mathematics dialogue weight-only methods MT-Bench