title: ThinK: Thinner Key Cache by Query-Driven Pruning
summary: Large Language Models (LLMs) have revolutionized the field of natural
  language processing, achieving unprecedented performance across a variety of
  applications by leveraging increased model sizes and sequence lengths. However,
  the associated rise in computational and memory costs poses significant
  challenges, particularly in managing long sequences due to the quadratic
  complexity of the transformer attention mechanism. This paper focuses on the
  long-context scenario, addressing the inefficiencies in KV cache memory
  consumption during inference. Unlike existing approaches that optimize the
  memory based on the sequence lengths, we uncover that the channel dimension of
  the KV cache exhibits significant redundancy, characterized by unbalanced
  magnitude distribution and low-rank structure in attention weights. Based on
  these observations, we propose ThinK, a novel query-dependent KV cache pruning
  method designed to minimize attention weight loss while selectively pruning the
  least significant channels. Our approach not only maintains or enhances model
  accuracy but also achieves a reduction in memory costs by over 20% compared
  with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and
  Mistral models across various long-sequence datasets confirm the efficacy of
  ThinK, setting a new precedent for efficient LLM deployment without
  compromising performance. We also outline the potential of extending our method
  to value cache pruning, demonstrating ThinK's versatility and broad
  applicability in reducing both memory and computational overheads.
paper_authors._id: 66a9b4a435c896924429516b
paper_authors.name: Yuhui Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-31T07:44:00.316Z
paper_authors._id: 6602869253a0518b2a98cafd
paper_authors.avatarUrl: /avatars/c14b5953a716f42c83ad28147f8308ae.svg
paper_authors.fullname: Yuhui Xu
paper_authors.type: user
paper_authors.user: yuhuixu
paper_authors._id: 66a9b4a435c896924429516c
paper_authors.name: Zhanming Jie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T07:58:57.282Z
paper_authors._id: 5f729da2f1e7ef6e919a1c27
paper_authors.avatarUrl: /avatars/bdf48f0d3290fab54d7e9636a3d14158.svg
paper_authors.fullname: Zhanming (Allan) Jie
paper_authors.type: user
paper_authors.user: allanjie
paper_authors._id: 66a9b4a435c896924429516d
paper_authors.name: Hanze Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T07:59:10.358Z
paper_authors._id: 63a3ff69f91ad3ea5703841d
paper_authors.avatarUrl: /avatars/69227c4bce01d33747c1377b6f9672db.svg
paper_authors.fullname: Hanze Dong
paper_authors.type: user
paper_authors.user: hendrydong
paper_authors._id: 66a9b4a435c896924429516e
paper_authors.name: Lei Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-31T12:49:35.556Z
paper_authors._id: 646def60df618b303b419323
paper_authors.avatarUrl: /avatars/97aa761d5255abf230304cfeade87835.svg
paper_authors.fullname: Lei Wang
paper_authors.type: user
paper_authors.user: demolei
paper_authors._id: 66a9b4a435c896924429516f
paper_authors.name: Xudong Lu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-31T07:44:02.548Z
paper_authors._id: 642e686bbe01b88c9446db8b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642e686bbe01b88c9446db8b/tb1DKe5xt50ykOeXiUuTE.jpeg
paper_authors.fullname: Lu Xudong
paper_authors.type: user
paper_authors.user: lucky-lance
paper_authors._id: 66a9b4a435c8969244295170
paper_authors.name: Aojun Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-11T07:45:31.264Z
paper_authors._id: 637de1520d5bb06fbe5207a9
paper_authors.avatarUrl: /avatars/1090851217270c5a858b13e013356d4f.svg
paper_authors.fullname: AJ.Zhou
paper_authors.type: user
paper_authors.user: AJZhou
paper_authors._id: 66a9b4a435c8969244295171
paper_authors.name: Amrita Saha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:01:09.678Z
paper_authors._id: 6461c2905dba83471db3be53
paper_authors.avatarUrl: /avatars/6e36cf86201d590ac729a75d4a439cde.svg
paper_authors.fullname: Amrita Saha
paper_authors.type: user
paper_authors.user: amritasaha87
paper_authors._id: 66a9b4a435c8969244295172
paper_authors.name: Caiming Xiong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:01:15.606Z
paper_authors._id: 649dbcc4e0fff1ed099dc80a
paper_authors.avatarUrl: /avatars/c87c273ca628dbcddccbf1ee19b2ce33.svg
paper_authors.fullname: Caiming Xiong
paper_authors.type: user
paper_authors.user: cxiong
paper_authors._id: 66a9b4a435c8969244295173
paper_authors.name: Doyen Sahoo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-31T08:01:22.415Z
paper_authors._id: 65f84fd980481173afd91233
paper_authors.avatarUrl: /avatars/6ac7bd6beba24d1476c5179b88c9e3fa.svg
paper_authors.fullname: Doyen
paper_authors.type: user
paper_authors.user: doyensahoo
paper_title: ThinK: Thinner Key Cache by Query-Driven Pruning
paper_summary: Large Language Models (LLMs) have revolutionized the field of natural
  language processing, achieving unprecedented performance across a variety of
  applications by leveraging increased model sizes and sequence lengths. However,
  the associated rise in computational and memory costs poses significant
  challenges, particularly in managing long sequences due to the quadratic
  complexity of the transformer attention mechanism. This paper focuses on the
  long-context scenario, addressing the inefficiencies in KV cache memory
  consumption during inference. Unlike existing approaches that optimize the
  memory based on the sequence lengths, we uncover that the channel dimension of
  the KV cache exhibits significant redundancy, characterized by unbalanced
  magnitude distribution and low-rank structure in attention weights. Based on
  these observations, we propose ThinK, a novel query-dependent KV cache pruning
  method designed to minimize attention weight loss while selectively pruning the
  least significant channels. Our approach not only maintains or enhances model
  accuracy but also achieves a reduction in memory costs by over 20% compared
  with vanilla KV cache eviction methods. Extensive evaluations on the LLaMA3 and
  Mistral models across various long-sequence datasets confirm the efficacy of
  ThinK, setting a new precedent for efficient LLM deployment without
  compromising performance. We also outline the potential of extending our method
  to value cache pruning, demonstrating ThinK's versatility and broad
  applicability in reducing both memory and computational overheads.
paper_ai_keywords: Large Language Models (LLMs) natural language processing sequence lengths transformer attention mechanism KV cache memory channel dimension unbalanced magnitude distribution low-rank structure attention weights query-dependent KV cache pruning attention weight loss memory costs model accuracy vanilla KV cache eviction methods LLaMA3 Mistral-vmodels long-sequence datasets value cache pruning computational overheads