A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language
  Models: An Experimental Analysis up to 405B
Prior research works have evaluated quantized LLMs using limited metrics such
as perplexity or a few basic knowledge tasks and old datasets. Additionally,
recent large-scale models such as Llama 3.1 with up to 405B have not been
thoroughly examined. This paper evaluates the performance of instruction-tuned
LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on
models ranging from 7B to 405B. Using 13 benchmarks, we assess performance
across six task types: commonsense Q\&A, knowledge and language understanding,
instruction following, hallucination detection, mathematics, and dialogue. Our
key findings reveal that (1) quantizing a larger LLM to a similar size as a
smaller FP16 LLM generally performs better across most benchmarks, except for
hallucination detection and instruction following; (2) performance varies
significantly with different quantization methods, model size, and bit-width,
with weight-only methods often yielding better results in larger models; (3)
task difficulty does not significantly impact accuracy degradation due to
quantization; and (4) the MT-Bench evaluation method has limited discriminatory
power among recent high-performing LLMs.
66ea30ceaeac53376f3b11d1
Jemin Lee
claimed_verified
2024-09-18T07:01:44.887Z
65b9dee19c4955ae7aee4954
/avatars/263f129605c7763185c49076174b891b.svg
Jemin Lee
user
leejaymin
66ea30ceaeac53376f3b11d2
Sihyeong Park
66ea30ceaeac53376f3b11d3
Jinse Kwon
admin_assigned
2024-09-18T08:08:41.717Z
65d6b1d0dee9f442f8ebf205
/avatars/9792889183b8e2718752ebed4fd1fcac.svg
Jinse Kwon
user
kwonse
66ea30ceaeac53376f3b11d4
Jihun Oh
admin_assigned
2024-09-18T08:09:04.894Z
63620c69da0599dc08cb200b
/avatars/9a6a035684e3765efd24517b81e241c5.svg
jihun oh
user
oj9040
66ea30ceaeac53376f3b11d5
Yongin Kwon
A Comprehensive Evaluation of Quantized Instruction-Tuned Large Language
  Models: An Experimental Analysis up to 405B
Prior research works have evaluated quantized LLMs using limited metrics such
as perplexity or a few basic knowledge tasks and old datasets. Additionally,
recent large-scale models such as Llama 3.1 with up to 405B have not been
thoroughly examined. This paper evaluates the performance of instruction-tuned
LLMs across various quantization methods (GPTQ, AWQ, SmoothQuant, and FP8) on
models ranging from 7B to 405B. Using 13 benchmarks, we assess performance
across six task types: commonsense Q\&A, knowledge and language understanding,
instruction following, hallucination detection, mathematics, and dialogue. Our
key findings reveal that (1) quantizing a larger LLM to a similar size as a
smaller FP16 LLM generally performs better across most benchmarks, except for
hallucination detection and instruction following; (2) performance varies
significantly with different quantization methods, model size, and bit-width,
with weight-only methods often yielding better results in larger models; (3)
task difficulty does not significantly impact accuracy degradation due to
quantization; and (4) the MT-Bench evaluation method has limited discriminatory
power among recent high-performing LLMs.
Various quantization methods are evaluated on large instruction-tuned LLMs across multiple benchmarks, revealing that quantized larger models often perform better than smaller FP16 models, except in some specific tasks.
quantized LLMs GPTQ AWQ SmoothQuant FP8 instruction-tuned LLMs commonsense Q&A knowledge and language understanding instruction following hallucination detection mathematics dialogue weight-only methods MT-Bench