title: Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV
    Cache Compression
summary: Visual Autoregressive (VAR) modeling has garnered significant attention for
  its innovative next-scale prediction approach, which yields substantial
  improvements in efficiency, scalability, and zero-shot generalization.
  Nevertheless, the coarse-to-fine methodology inherent in VAR results in
  exponential growth of the KV cache during inference, causing considerable
  memory consumption and computational redundancy. To address these bottlenecks,
  we introduce ScaleKV, a novel KV cache compression framework tailored for VAR
  architectures. ScaleKV leverages two critical observations: varying cache
  demands across transformer layers and distinct attention patterns at different
  scales. Based on these insights, ScaleKV categorizes transformer layers into
  two functional groups: drafters and refiners. Drafters exhibit dispersed
  attention across multiple scales, thereby requiring greater cache capacity.
  Conversely, refiners focus attention on the current token map to process local
  details, consequently necessitating substantially reduced cache capacity.
  ScaleKV optimizes the multi-scale inference pipeline by identifying
  scale-specific drafters and refiners, facilitating differentiated cache
  management tailored to each scale. Evaluation on the state-of-the-art
  text-to-image VAR model family, Infinity, demonstrates that our approach
  effectively reduces the required KV cache memory to 10% while preserving
  pixel-level fidelity.
paper_authors._id: 6835239e7309025530c85ba3
paper_authors.name: Kunjun Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:51:58.885Z
paper_authors._id: 6540ef0e733c1ce6a6fc989a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6540ef0e733c1ce6a6fc989a/lyDLbmJ-h4nUmkWZCvWtg.jpeg
paper_authors.fullname: Kunjun Li
paper_authors.type: user
paper_authors.user: stargazerx0
paper_authors._id: 6835239e7309025530c85ba4
paper_authors.name: Zigeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T15:50:12.945Z
paper_authors._id: 65811eeaa2284a018e51f1ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
paper_authors.fullname: Zigeng Chen
paper_authors.type: user
paper_authors.user: Zigeng
paper_authors._id: 6835239e7309025530c85ba5
paper_authors.name: Cheng-Yen Yang
paper_authors._id: 6835239e7309025530c85ba6
paper_authors.name: Jenq-Neng Hwang
paper_title: Memory-Efficient Visual Autoregressive Modeling with Scale-Aware KV
    Cache Compression
paper_summary: Visual Autoregressive (VAR) modeling has garnered significant attention for
  its innovative next-scale prediction approach, which yields substantial
  improvements in efficiency, scalability, and zero-shot generalization.
  Nevertheless, the coarse-to-fine methodology inherent in VAR results in
  exponential growth of the KV cache during inference, causing considerable
  memory consumption and computational redundancy. To address these bottlenecks,
  we introduce ScaleKV, a novel KV cache compression framework tailored for VAR
  architectures. ScaleKV leverages two critical observations: varying cache
  demands across transformer layers and distinct attention patterns at different
  scales. Based on these insights, ScaleKV categorizes transformer layers into
  two functional groups: drafters and refiners. Drafters exhibit dispersed
  attention across multiple scales, thereby requiring greater cache capacity.
  Conversely, refiners focus attention on the current token map to process local
  details, consequently necessitating substantially reduced cache capacity.
  ScaleKV optimizes the multi-scale inference pipeline by identifying
  scale-specific drafters and refiners, facilitating differentiated cache
  management tailored to each scale. Evaluation on the state-of-the-art
  text-to-image VAR model family, Infinity, demonstrates that our approach
  effectively reduces the required KV cache memory to 10% while preserving
  pixel-level fidelity.
paper_ai_summary: ScaleKV compresses the KV cache in Visual Autoregressive models by differentiating drafters and refiners across transformer layers, reducing memory consumption while maintaining high fidelity.
paper_ai_keywords: Visual Autoregressive VAR KV cache transformer layers drafters refiners memory consumption Infinity pixel-level fidelity