title: Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
summary: The growing demand for Large Language Models (LLMs) in applications such as
  content generation, intelligent chatbots, and sentiment analysis poses
  considerable challenges for LLM service providers. To efficiently use GPU
  resources and boost throughput, batching multiple requests has emerged as a
  popular paradigm; to further speed up batching, LLM quantization techniques
  reduce memory consumption and increase computing capacity. However, prevalent
  quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully
  leverage the capabilities of modern GPUs, such as 4-bit integer operators,
  resulting in sub-optimal performance.
    To maximize LLMs' serving throughput, we introduce Atom, a low-bit
  quantization method that achieves high throughput improvements with negligible
  accuracy loss. Atom significantly boosts serving throughput by using low-bit
  operators and considerably reduces memory consumption via low-bit quantization.
  It attains high accuracy by applying a novel mixed-precision and fine-grained
  quantization process. We evaluate Atom on 4-bit weight-activation quantization
  setups in the serving context. Atom improves end-to-end throughput by up to
  7.73times compared to the FP16 and by 2.53times compared to INT8
  quantization, while maintaining the same latency target.
paper_authors._id: 65408d341c4b203c012efcde
paper_authors.name: Yilong Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-07T11:20:52.551Z
paper_authors._id: 6549b0a808775ce78e535c6a
paper_authors.avatarUrl: /avatars/942066356843d0c424375937f157c975.svg
paper_authors.fullname: Yilong Zhao
paper_authors.type: user
paper_authors.user: ylzhao
paper_authors._id: 65408d341c4b203c012efcdf
paper_authors.name: Chien-Yu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:33:33.415Z
paper_authors._id: 64d1950a0fbfb00b919e9481
paper_authors.avatarUrl: /avatars/7a5f367882012bd08a3ca73f9acd9912.svg
paper_authors.fullname: Chien-Yu Lin
paper_authors.type: user
paper_authors.user: cylinbao
paper_authors._id: 65408d341c4b203c012efce0
paper_authors.name: Kan Zhu
paper_authors._id: 65408d341c4b203c012efce1
paper_authors.name: Zihao Ye
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:33:49.784Z
paper_authors._id: 632b9f58ffe3618eff2ffb93
paper_authors.avatarUrl: /avatars/1ffaf2da94fb9f993aa0af4bc3808197.svg
paper_authors.fullname: Zihao Ye
paper_authors.type: user
paper_authors.user: zhye
paper_authors._id: 65408d341c4b203c012efce2
paper_authors.name: Lequn Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:33:56.655Z
paper_authors._id: 63f6a8f9e94ed998c460302a
paper_authors.avatarUrl: /avatars/c2b78e2725a5c8c1cd3ab2e502afff71.svg
paper_authors.fullname: Lequn Chen
paper_authors.type: user
paper_authors.user: abcdabcd987
paper_authors._id: 65408d341c4b203c012efce3
paper_authors.name: Size Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-31T10:34:03.611Z
paper_authors._id: 636c5cb1d963c567935dd7a7
paper_authors.avatarUrl: /avatars/bfe016cd8f644039f21ed36163d36cae.svg
paper_authors.fullname: Size Zheng
paper_authors.type: user
paper_authors.user: ZCHNO
paper_authors._id: 65408d341c4b203c012efce4
paper_authors.name: Luis Ceze
paper_authors._id: 65408d341c4b203c012efce5
paper_authors.name: Arvind Krishnamurthy
paper_authors._id: 65408d341c4b203c012efce6
paper_authors.name: Tianqi Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T15:04:46.324Z
paper_authors._id: 6407c3cb2e309e654528860f
paper_authors.avatarUrl: /avatars/b2177ae76ff8001564d0606752538884.svg
paper_authors.fullname: Tianqi Chen
paper_authors.type: user
paper_authors.user: tqchen
paper_authors._id: 65408d341c4b203c012efce7
paper_authors.name: Baris Kasikci
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-31T17:02:05.536Z
paper_authors._id: 654132fe5a9a913c6c870e79
paper_authors.avatarUrl: /avatars/2f6807eddef1929c571977e9af35f952.svg
paper_authors.fullname: Baris Kasikci
paper_authors.type: user
paper_authors.user: kasikci
paper_title: Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
paper_summary: The growing demand for Large Language Models (LLMs) in applications such as
  content generation, intelligent chatbots, and sentiment analysis poses
  considerable challenges for LLM service providers. To efficiently use GPU
  resources and boost throughput, batching multiple requests has emerged as a
  popular paradigm; to further speed up batching, LLM quantization techniques
  reduce memory consumption and increase computing capacity. However, prevalent
  quantization schemes (e.g., 8-bit weight-activation quantization) cannot fully
  leverage the capabilities of modern GPUs, such as 4-bit integer operators,
  resulting in sub-optimal performance.
    To maximize LLMs' serving throughput, we introduce Atom, a low-bit
  quantization method that achieves high throughput improvements with negligible
  accuracy loss. Atom significantly boosts serving throughput by using low-bit
  operators and considerably reduces memory consumption via low-bit quantization.
  It attains high accuracy by applying a novel mixed-precision and fine-grained
  quantization process. We evaluate Atom on 4-bit weight-activation quantization
  setups in the serving context. Atom improves end-to-end throughput by up to
  7.73times compared to the FP16 and by 2.53times compared to INT8
  quantization, while maintaining the same latency target.
paper_ai_summary: Atom is a low-bit quantization method that enhances LLM serving throughput with minimal accuracy loss by utilizing low-bit operators and a mixed-precision process.
paper_ai_keywords: Large Language Models LLMs GPU batching quantization memory consumption 4-bit integer operators serving throughput low-bit operators mixed-precision fine-grained quantization FP16 INT8 quantization latency target