title: LinFusion: 1 GPU, 1 Minute, 16K Image
summary: Modern diffusion models, particularly those utilizing a Transformer-based
  UNet for denoising, rely heavily on self-attention operations to manage complex
  spatial relationships, thus achieving impressive generation performance.
  However, this existing paradigm faces significant challenges in generating
  high-resolution visual content due to its quadratic time and memory complexity
  with respect to the number of spatial tokens. To address this limitation, we
  aim at a novel linear attention mechanism as an alternative in this paper.
  Specifically, we begin our exploration from recently introduced models with
  linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and
  identify two key features-attention normalization and non-causal inference-that
  enhance high-resolution visual generation performance. Building on these
  insights, we introduce a generalized linear attention paradigm, which serves as
  a low-rank approximation of a wide spectrum of popular linear token mixers. To
  save the training cost and better leverage pre-trained models, we initialize
  our models and distill the knowledge from pre-trained StableDiffusion (SD). We
  find that the distilled model, termed LinFusion, achieves performance on par
  with or superior to the original SD after only modest training, while
  significantly reducing time and memory complexity. Extensive experiments on
  SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory
  zero-shot cross-resolution generation performance, generating high-resolution
  images like 16K resolution. Moreover, it is highly compatible with pre-trained
  SD components, such as ControlNet and IP-Adapter, requiring no adaptation
  efforts. Codes are available at https://github.com/Huage001/LinFusion.
paper_authors._id: 66d7ca340d88366001a864d0
paper_authors.name: Songhua Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:53:11.962Z
paper_authors._id: 64b929308b53fb5dbd059ce3
paper_authors.avatarUrl: /avatars/564c44cf45db794747a96f79f30ecd91.svg
paper_authors.fullname: Liu Songhua
paper_authors.type: user
paper_authors.user: Huage001
paper_authors._id: 66d7ca340d88366001a864d1
paper_authors.name: Weihao Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:53:42.668Z
paper_authors._id: 5df833bdda6d0311fd3d5403
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
paper_authors.fullname: Weihao Yu
paper_authors.type: user
paper_authors.user: whyu
paper_authors._id: 66d7ca340d88366001a864d2
paper_authors.name: Zhenxiong Tan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-04T08:53:53.422Z
paper_authors._id: 664ef197d3f8759dddfbc765
paper_authors.avatarUrl: /avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
paper_authors.fullname: Tan Zhen Xiong
paper_authors.type: user
paper_authors.user: ZhenXiong
paper_authors._id: 66d7ca340d88366001a864d3
paper_authors.name: Xinchao Wang
paper_title: LinFusion: 1 GPU, 1 Minute, 16K Image
paper_summary: Modern diffusion models, particularly those utilizing a Transformer-based
  UNet for denoising, rely heavily on self-attention operations to manage complex
  spatial relationships, thus achieving impressive generation performance.
  However, this existing paradigm faces significant challenges in generating
  high-resolution visual content due to its quadratic time and memory complexity
  with respect to the number of spatial tokens. To address this limitation, we
  aim at a novel linear attention mechanism as an alternative in this paper.
  Specifically, we begin our exploration from recently introduced models with
  linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and
  identify two key features-attention normalization and non-causal inference-that
  enhance high-resolution visual generation performance. Building on these
  insights, we introduce a generalized linear attention paradigm, which serves as
  a low-rank approximation of a wide spectrum of popular linear token mixers. To
  save the training cost and better leverage pre-trained models, we initialize
  our models and distill the knowledge from pre-trained StableDiffusion (SD). We
  find that the distilled model, termed LinFusion, achieves performance on par
  with or superior to the original SD after only modest training, while
  significantly reducing time and memory complexity. Extensive experiments on
  SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory
  zero-shot cross-resolution generation performance, generating high-resolution
  images like 16K resolution. Moreover, it is highly compatible with pre-trained
  SD components, such as ControlNet and IP-Adapter, requiring no adaptation
  efforts. Codes are available at https://github.com/Huage001/LinFusion.
paper_ai_summary: A novel linear attention mechanism, named LinFusion, is introduced to enhance high-resolution image generation with reduced computational cost, offering performance comparable to or better than existing diffusion models.
paper_ai_keywords: diffusion models Transformer-based UNet self-attention operations linear attention mechanism Mamba Mamba2 Gated Linear Attention attention normalization non-causal inference generalized linear attention paradigm low-rank approximation linear token mixers StableDiffusion (SD) zero-shot cross-resolution generation 16K resolution ControlNet IP-Adapter