title: TP-Aware Dequantization
summary: In this paper, we present a novel method that reduces model inference latency
  during distributed deployment of Large Language Models (LLMs). Our contribution
  is an optimized inference deployment scheme that address the current
  limitations of state-of-the-art quantization kernels when used in conjunction
  with Tensor Parallel (TP). Our method preserves data locality in GPU memory
  access patterns and exploits a priori knowledge of TP to reduce global
  communication. We demonstrate an up to 1.81x speedup over existing methods for
  Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
  problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
paper_authors._id: 65c453ddb25522fdb01dd2bd
paper_authors.name: Adnan Hoque
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-02-08T04:09:01.800Z
paper_authors._id: 65c2782802e7932529fae7b3
paper_authors.avatarUrl: /avatars/082e37e2b1bba64f7b41ef52ce32b642.svg
paper_authors.fullname: Adnan Hoque
paper_authors.type: user
paper_authors.user: adhoq26
paper_authors._id: 65c453ddb25522fdb01dd2be
paper_authors.name: Mudhakar Srivatsa
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T12:05:16.906Z
paper_authors._id: 650887ccf75ac8c6f958d434
paper_authors.avatarUrl: /avatars/6dff3ded603e1975425bb481d0e63380.svg
paper_authors.fullname: Mudhakar Srivatsa
paper_authors.type: user
paper_authors.user: mudhakar
paper_authors._id: 65c453ddb25522fdb01dd2bf
paper_authors.name: Chih-Chieh Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T12:05:23.656Z
paper_authors._id: 646b38a8ec9a61e8717aaec5
paper_authors.avatarUrl: /avatars/c2d6d4ee1f36889170efdfcb46425014.svg
paper_authors.fullname: Chih-Chieh Yang
paper_authors.type: user
paper_authors.user: scatjay
paper_authors._id: 65c453ddb25522fdb01dd2c0
paper_authors.name: Raghu Ganti
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T12:05:30.590Z
paper_authors._id: 61d460b42e1399aebcf80a39
paper_authors.avatarUrl: /avatars/7eb33d86f5373a4b3c16b72bfbd4cb0d.svg
paper_authors.fullname: Raghu Ganti
paper_authors.type: user
paper_authors.user: rganti
paper_title: TP-Aware Dequantization
paper_summary: In this paper, we present a novel method that reduces model inference latency
  during distributed deployment of Large Language Models (LLMs). Our contribution
  is an optimized inference deployment scheme that address the current
  limitations of state-of-the-art quantization kernels when used in conjunction
  with Tensor Parallel (TP). Our method preserves data locality in GPU memory
  access patterns and exploits a priori knowledge of TP to reduce global
  communication. We demonstrate an up to 1.81x speedup over existing methods for
  Llama-70B and up to 1.78x speedup for IBM WatsonX's Granite-20B MLP layer
  problem sizes on A100 and H100 NVIDIA DGX Systems for a variety of TP settings.
paper_ai_keywords: Large Language Models (LLMs) Tensor Parallel (TP) data locality global communication quantization kernels Llama-70B Granite-20B MLP layer A100 H100 NVIDIA DGX Systems