title: "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
    Quantization
summary: Despite the popularity of large language model (LLM) quantization for
  inference acceleration, significant uncertainty remains regarding the
  accuracy-performance trade-offs associated with various quantization formats.
  We present a comprehensive empirical study of quantized accuracy, evaluating
  popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
  real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
  examines the difference in text generated by quantized models versus their
  uncompressed counterparts. Beyond benchmarks, we also present a couple of
  quantization improvements which allowed us to obtain state-of-the-art accuracy
  recovery results. Our investigation, encompassing over 500,000 individual
  evaluations, yields several key findings: (1) FP8 weight and activation
  quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
  activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
  low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
  (W4A16-INT) is competitive with 8-bit integer weight and activation
  quantization. To address the question of the "best" format for a given
  deployment environment, we conduct inference performance analysis using the
  popular open-source vLLM framework on various GPU architectures. We find that
  W4A16 offers the best cost-efficiency for synchronous deployments, and for
  asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
  in asynchronous "continuous batching" deployment of mid- and large-size models
  on high-end GPUs. Our results provide a set of practical guidelines for
  deploying quantized LLMs across scales and performance requirements.
paper_authors._id: 672a1fde369f174c868341ec
paper_authors.name: Eldar Kurtic
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T13:41:42.690Z
paper_authors._id: 628e0ce4e53bbd334577fcb0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668801593252-628e0ce4e53bbd334577fcb0.jpeg
paper_authors.fullname: Eldar Kurtic
paper_authors.type: user
paper_authors.user: ekurtic
paper_authors._id: 672a1fde369f174c868341ed
paper_authors.name: Alexandre Marques
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:46:19.705Z
paper_authors._id: 64ee5432f9ed75901c09a590
paper_authors.avatarUrl: /avatars/992163fe824381af39cedec073136967.svg
paper_authors.fullname: Alexandre Marques
paper_authors.type: user
paper_authors.user: alexmarques
paper_authors._id: 672a1fde369f174c868341ee
paper_authors.name: Shubhra Pandit
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:45:54.783Z
paper_authors._id: 632c828a54e2c512c8f55144
paper_authors.avatarUrl: /avatars/a30e4b33b21750833a38ab44a874a4db.svg
paper_authors.fullname: Shubhra Pandit
paper_authors.type: user
paper_authors.user: shubhrapandit
paper_authors._id: 672a1fde369f174c868341ef
paper_authors.name: Mark Kurtz
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-05T20:17:30.459Z
paper_authors._id: 628e965178e24401298387ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/628e965178e24401298387ee/IzynaZqiVA6X27MKFiKmF.jpeg
paper_authors.fullname: Mark Kurtz
paper_authors.type: user
paper_authors.user: markurtz
paper_authors._id: 672a1fde369f174c868341f0
paper_authors.name: Dan Alistarh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-05T14:45:08.138Z
paper_authors._id: 64d100c5d8d0927372e3d4c0
paper_authors.avatarUrl: /avatars/91d9e4f1dab25b70d901783cdfcd2fd1.svg
paper_authors.fullname: Dan Alistarh
paper_authors.type: user
paper_authors.user: dalistarh
paper_title: "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM
    Quantization
paper_summary: Despite the popularity of large language model (LLM) quantization for
  inference acceleration, significant uncertainty remains regarding the
  accuracy-performance trade-offs associated with various quantization formats.
  We present a comprehensive empirical study of quantized accuracy, evaluating
  popular quantization formats (FP8, INT8, INT4) across academic benchmarks and
  real-world tasks, on the entire Llama-3.1 model family. Additionally, our study
  examines the difference in text generated by quantized models versus their
  uncompressed counterparts. Beyond benchmarks, we also present a couple of
  quantization improvements which allowed us to obtain state-of-the-art accuracy
  recovery results. Our investigation, encompassing over 500,000 individual
  evaluations, yields several key findings: (1) FP8 weight and activation
  quantization (W8A8-FP) is lossless across all model scales, (2) INT8 weight and
  activation quantization (W8A8-INT), when properly tuned, incurs surprisingly
  low 1-3% accuracy degradation, and (3) INT4 weight-only quantization
  (W4A16-INT) is competitive with 8-bit integer weight and activation
  quantization. To address the question of the "best" format for a given
  deployment environment, we conduct inference performance analysis using the
  popular open-source vLLM framework on various GPU architectures. We find that
  W4A16 offers the best cost-efficiency for synchronous deployments, and for
  asynchronous deployment on mid-tier GPUs. At the same time, W8A8 formats excel
  in asynchronous "continuous batching" deployment of mid- and large-size models
  on high-end GPUs. Our results provide a set of practical guidelines for
  deploying quantized LLMs across scales and performance requirements.
paper_ai_keywords: large language model (LLM) quantization inference acceleration quantization formats FP8 INT8 quantization Llama-3.1 model family text generation W8A8-FP W8A8-INT W4A16-INT vLLM framework synchronous deployments asynchronous deployment mid-tier GPUs continuous batching high-end GPUs quantized LLMs