title: Efficient Inference of Vision Instruction-Following Models with Elastic
    Cache
summary: In the field of instruction-following large vision-language models (LVLMs),
  the efficient deployment of these models faces challenges, notably due to the
  high memory demands of their key-value (KV) caches. Conventional cache
  management strategies for LLMs focus on cache eviction, which often fails to
  address the specific needs of multimodal instruction-following models.
  Recognizing this gap, in this paper, we introduce Elastic Cache, a novel
  approach that benefits from applying distinct acceleration methods for
  instruction encoding and output generation stages. We investigate the metrics
  of importance in different stages and propose an importance-driven cache
  merging strategy to prune redundancy caches. Instead of discarding less
  important caches, our strategy identifies important key/value vectors as anchor
  points. Surrounding less important caches are then merged with these anchors,
  enhancing the preservation of contextual information in the KV caches while
  yielding an arbitrary acceleration ratio. For instruction encoding, we utilize
  the frequency to evaluate the importance of caches. Regarding output
  generation, we prioritize tokens based on their distance with an offset, by
  which both the initial and most recent tokens are retained. Results on a range
  of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also
  notably outperforms existing pruning methods in language generation across
  various tasks. Code is available at https://github.com/liuzuyan/ElasticCache
paper_authors._id: 66a3aa35ee3de8c56edca432
paper_authors.name: Zuyan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T14:03:37.394Z
paper_authors._id: 64f001bfabd9fb1914398bd5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64f001bfabd9fb1914398bd5/9teH82hkBI4csIz_WQh5q.jpeg
paper_authors.fullname: liuzuyan
paper_authors.type: user
paper_authors.user: Zuyan
paper_authors._id: 66a3aa35ee3de8c56edca433
paper_authors.name: Benlin Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-02T12:23:18.285Z
paper_authors._id: 6326723c21c98f83d6ad128e
paper_authors.avatarUrl: /avatars/6c208ff815c037d0a6a2ada3198e7fd4.svg
paper_authors.fullname: Benlin Liu
paper_authors.type: user
paper_authors.user: Tim666
paper_authors._id: 66a3aa35ee3de8c56edca434
paper_authors.name: Jiahui Wang
paper_authors._id: 66a3aa35ee3de8c56edca435
paper_authors.name: Yuhao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T14:06:03.783Z
paper_authors._id: 652965773a416e1f2173443b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/652965773a416e1f2173443b/y9MB8YgHzbwCXAc4EI9T3.jpeg
paper_authors.fullname: Yuhao Dong
paper_authors.type: user
paper_authors.user: THUdyh
paper_authors._id: 66a3aa35ee3de8c56edca436
paper_authors.name: Guangyi Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T14:06:11.477Z
paper_authors._id: 64975e18a08b5c7ea0163be7
paper_authors.avatarUrl: /avatars/88963b6ae00ec3f10f4a40322f75944a.svg
paper_authors.fullname: Guangyi Chen
paper_authors.type: user
paper_authors.user: CHENGY12
paper_authors._id: 66a3aa35ee3de8c56edca437
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T14:06:19.070Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_authors._id: 66a3aa35ee3de8c56edca438
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-26T14:06:25.297Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 66a3aa35ee3de8c56edca439
paper_authors.name: Jiwen Lu
paper_title: Efficient Inference of Vision Instruction-Following Models with Elastic
    Cache
paper_summary: In the field of instruction-following large vision-language models (LVLMs),
  the efficient deployment of these models faces challenges, notably due to the
  high memory demands of their key-value (KV) caches. Conventional cache
  management strategies for LLMs focus on cache eviction, which often fails to
  address the specific needs of multimodal instruction-following models.
  Recognizing this gap, in this paper, we introduce Elastic Cache, a novel
  approach that benefits from applying distinct acceleration methods for
  instruction encoding and output generation stages. We investigate the metrics
  of importance in different stages and propose an importance-driven cache
  merging strategy to prune redundancy caches. Instead of discarding less
  important caches, our strategy identifies important key/value vectors as anchor
  points. Surrounding less important caches are then merged with these anchors,
  enhancing the preservation of contextual information in the KV caches while
  yielding an arbitrary acceleration ratio. For instruction encoding, we utilize
  the frequency to evaluate the importance of caches. Regarding output
  generation, we prioritize tokens based on their distance with an offset, by
  which both the initial and most recent tokens are retained. Results on a range
  of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also
  notably outperforms existing pruning methods in language generation across
  various tasks. Code is available at https://github.com/liuzuyan/ElasticCache
paper_ai_keywords: key-value (KV) caches Elastic Cache importance-driven cache merging strategy anchor points contextual information frequency offset language generation