title: SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
summary: Key-Value (KV) cache has become a bottleneck of LLMs for long-context
  generation. Despite the numerous efforts in this area, the optimization for the
  decoding phase is generally ignored. However, we believe such optimization is
  crucial, especially for long-output generation tasks based on the following two
  observations: (i) Excessive compression during the prefill phase, which
  requires specific full context impairs the comprehension of the reasoning task;
  (ii) Deviation of heavy hitters occurs in the reasoning tasks with long
  outputs. Therefore, SCOPE, a simple yet efficient framework that separately
  performs KV cache optimization during the prefill and decoding phases, is
  introduced. Specifically, the KV cache during the prefill phase is preserved to
  maintain the essential information, while a novel strategy based on sliding is
  proposed to select essential heavy hitters for the decoding phase. Memory usage
  and memory transfer are further optimized using adaptive and discontinuous
  strategies. Extensive experiments on LongGenBench show the effectiveness and
  generalization of SCOPE and its compatibility as a plug-in to other
  prefill-only KV compression methods.
paper_authors._id: 6768cd61aa9027defefa2ad4
paper_authors.name: Jialong Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-23T11:10:00.992Z
paper_authors._id: 644a4fbc2166258fccc664bc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/8k3b44MbhQiWuo6i8BnYl.jpeg
paper_authors.fullname: Jialong Wu
paper_authors.type: user
paper_authors.user: callanwu
paper_authors._id: 6768cd61aa9027defefa2ad5
paper_authors.name: Zhenglin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-30T19:36:19.930Z
paper_authors._id: 6643261b8876db14227eeb19
paper_authors.avatarUrl: /avatars/67428c9e37a2273697c0547e1783ec6b.svg
paper_authors.fullname: Zhenglin Wang
paper_authors.type: user
paper_authors.user: wzl0228
paper_authors._id: 6768cd61aa9027defefa2ad6
paper_authors.name: Linhai Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:30:58.318Z
paper_authors._id: 66596d64ce1b2838888f4401
paper_authors.avatarUrl: /avatars/d8d0d116a3198571c7e86f09871c2d76.svg
paper_authors.fullname: Linhai Zhang
paper_authors.type: user
paper_authors.user: lzhang472
paper_authors._id: 6768cd61aa9027defefa2ad7
paper_authors.name: Yilong Lai
paper_authors._id: 6768cd61aa9027defefa2ad8
paper_authors.name: Yulan He
paper_authors._id: 6768cd61aa9027defefa2ad9
paper_authors.name: Deyu Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-23T11:31:24.902Z
paper_authors._id: 64e821f2bddc5b1072b15c2e
paper_authors.avatarUrl: /avatars/618b5a48f2fa62daff4e1922a9aa9e8b.svg
paper_authors.fullname: zhoudeyu
paper_authors.type: user
paper_authors.user: zhoudeyu
paper_title: SCOPE: Optimizing Key-Value Cache Compression in Long-context Generation
paper_summary: Key-Value (KV) cache has become a bottleneck of LLMs for long-context
  generation. Despite the numerous efforts in this area, the optimization for the
  decoding phase is generally ignored. However, we believe such optimization is
  crucial, especially for long-output generation tasks based on the following two
  observations: (i) Excessive compression during the prefill phase, which
  requires specific full context impairs the comprehension of the reasoning task;
  (ii) Deviation of heavy hitters occurs in the reasoning tasks with long
  outputs. Therefore, SCOPE, a simple yet efficient framework that separately
  performs KV cache optimization during the prefill and decoding phases, is
  introduced. Specifically, the KV cache during the prefill phase is preserved to
  maintain the essential information, while a novel strategy based on sliding is
  proposed to select essential heavy hitters for the decoding phase. Memory usage
  and memory transfer are further optimized using adaptive and discontinuous
  strategies. Extensive experiments on LongGenBench show the effectiveness and
  generalization of SCOPE and its compatibility as a plug-in to other
  prefill-only KV compression methods.
paper_ai_summary: SCOPE is a framework that optimizes KV cache separately during prefill and decoding phases to improve long-context and long-output generation in LLMs.
paper_ai_keywords: KV cache LLMs long-context generation optimization decoding phase prefill phase full context heavy hitters LongGenBench adaptive strategies discontinuous strategies