title: Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
summary: In the rapidly advancing field of image generation, Visual Auto-Regressive
  (VAR) modeling has garnered considerable attention for its innovative
  next-scale prediction approach. This paradigm offers substantial improvements
  in efficiency, scalability, and zero-shot generalization. Yet, the inherently
  coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to
  prohibitive memory consumption and computational redundancies. To address these
  bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient
  decoding strategy tailored for the VAR framework. CoDe capitalizes on two
  critical observations: the substantially reduced parameter demands at larger
  scales and the exclusive generation patterns across different scales. Based on
  these insights, we partition the multi-scale inference process into a seamless
  collaboration between a large model and a small model. The large model serves
  as the 'drafter', specializing in generating low-frequency content at smaller
  scales, while the smaller model serves as the 'refiner', solely focusing on
  predicting high-frequency details at larger scales. This collaboration yields
  remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x
  speedup, slashes memory usage by around 50%, and preserves image quality with
  only a negligible FID increase from 1.95 to 1.98. When drafting steps are
  further decreased, CoDe can achieve an impressive 2.9x acceleration ratio,
  reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while
  preserving a commendable FID of 2.27. The code is available at
  https://github.com/czg1225/CoDe
paper_authors._id: 6747d429cd2486b8b0e1fb24
paper_authors.name: Zigeng Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:44.162Z
paper_authors._id: 65811eeaa2284a018e51f1ba
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/dH8UZj6Kk5HJkI1DItCNm.jpeg
paper_authors.fullname: Zigeng Chen
paper_authors.type: user
paper_authors.user: Zigeng
paper_authors._id: 6747d429cd2486b8b0e1fb25
paper_authors.name: Xinyin Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:42.545Z
paper_authors._id: 64396ebc21221ac7411852b3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64396ebc21221ac7411852b3/SR0dC8N0bdj9tZFxYPpSf.jpeg
paper_authors.fullname: Xinyin Ma
paper_authors.type: user
paper_authors.user: horseee
paper_authors._id: 6747d429cd2486b8b0e1fb26
paper_authors.name: Gongfan Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-16T13:42:16.302Z
paper_authors._id: 646a1939c37ca1e12308fe81
paper_authors.avatarUrl: /avatars/752e9d86018e7d33ad8bcd741203fd86.svg
paper_authors.fullname: Gongfan Fang
paper_authors.type: user
paper_authors.user: Vinnnf
paper_authors._id: 6747d429cd2486b8b0e1fb27
paper_authors.name: Xinchao Wang
paper_title: Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient
paper_summary: In the rapidly advancing field of image generation, Visual Auto-Regressive
  (VAR) modeling has garnered considerable attention for its innovative
  next-scale prediction approach. This paradigm offers substantial improvements
  in efficiency, scalability, and zero-shot generalization. Yet, the inherently
  coarse-to-fine nature of VAR introduces a prolonged token sequence, leading to
  prohibitive memory consumption and computational redundancies. To address these
  bottlenecks, we propose Collaborative Decoding (CoDe), a novel efficient
  decoding strategy tailored for the VAR framework. CoDe capitalizes on two
  critical observations: the substantially reduced parameter demands at larger
  scales and the exclusive generation patterns across different scales. Based on
  these insights, we partition the multi-scale inference process into a seamless
  collaboration between a large model and a small model. The large model serves
  as the 'drafter', specializing in generating low-frequency content at smaller
  scales, while the smaller model serves as the 'refiner', solely focusing on
  predicting high-frequency details at larger scales. This collaboration yields
  remarkable efficiency with minimal impact on quality: CoDe achieves a 1.7x
  speedup, slashes memory usage by around 50%, and preserves image quality with
  only a negligible FID increase from 1.95 to 1.98. When drafting steps are
  further decreased, CoDe can achieve an impressive 2.9x acceleration ratio,
  reaching 41 images/s at 256x256 resolution on a single NVIDIA 4090 GPU, while
  preserving a commendable FID of 2.27. The code is available at
  https://github.com/czg1225/CoDe
paper_ai_summary: Collaborative Decoding (CoDe) enhances Visual Auto-Regressive (VAR) modeling efficiency by integrating a large and small model to reduce memory usage and computation time without significantly affecting image quality.
paper_ai_keywords: Visual Auto-Regressive (VAR) modeling Collaborative Decoding (CoDe) multi-scale inference large model small model drafter refiner low-frequency content high-frequency details speedup memory usage FID NVIDIA 4090 GPU