LinFusion: 1 GPU, 1 Minute, 16K Image
Modern diffusion models, particularly those utilizing a Transformer-based
UNet for denoising, rely heavily on self-attention operations to manage complex
spatial relationships, thus achieving impressive generation performance.
However, this existing paradigm faces significant challenges in generating
high-resolution visual content due to its quadratic time and memory complexity
with respect to the number of spatial tokens. To address this limitation, we
aim at a novel linear attention mechanism as an alternative in this paper.
Specifically, we begin our exploration from recently introduced models with
linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and
identify two key features-attention normalization and non-causal inference-that
enhance high-resolution visual generation performance. Building on these
insights, we introduce a generalized linear attention paradigm, which serves as
a low-rank approximation of a wide spectrum of popular linear token mixers. To
save the training cost and better leverage pre-trained models, we initialize
our models and distill the knowledge from pre-trained StableDiffusion (SD). We
find that the distilled model, termed LinFusion, achieves performance on par
with or superior to the original SD after only modest training, while
significantly reducing time and memory complexity. Extensive experiments on
SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory
zero-shot cross-resolution generation performance, generating high-resolution
images like 16K resolution. Moreover, it is highly compatible with pre-trained
SD components, such as ControlNet and IP-Adapter, requiring no adaptation
efforts. Codes are available at https://github.com/Huage001/LinFusion.
66d7ca340d88366001a864d0
Songhua Liu
admin_assigned
2024-09-04T08:53:11.962Z
64b929308b53fb5dbd059ce3
/avatars/564c44cf45db794747a96f79f30ecd91.svg
Liu Songhua
user
Huage001
66d7ca340d88366001a864d1
Weihao Yu
admin_assigned
2024-09-04T08:53:42.668Z
5df833bdda6d0311fd3d5403
https://cdn-avatars.huggingface.co/v1/production/uploads/5df833bdda6d0311fd3d5403/62OtGJEQXdOuhV9yCd4HS.png
Weihao Yu
user
whyu
66d7ca340d88366001a864d2
Zhenxiong Tan
admin_assigned
2024-09-04T08:53:53.422Z
664ef197d3f8759dddfbc765
/avatars/fd3f8a844b391be6c1ac2ea89b228f5a.svg
Tan Zhen Xiong
user
ZhenXiong
66d7ca340d88366001a864d3
Xinchao Wang
LinFusion: 1 GPU, 1 Minute, 16K Image
Modern diffusion models, particularly those utilizing a Transformer-based
UNet for denoising, rely heavily on self-attention operations to manage complex
spatial relationships, thus achieving impressive generation performance.
However, this existing paradigm faces significant challenges in generating
high-resolution visual content due to its quadratic time and memory complexity
with respect to the number of spatial tokens. To address this limitation, we
aim at a novel linear attention mechanism as an alternative in this paper.
Specifically, we begin our exploration from recently introduced models with
linear complexity, e.g., Mamba, Mamba2, and Gated Linear Attention, and
identify two key features-attention normalization and non-causal inference-that
enhance high-resolution visual generation performance. Building on these
insights, we introduce a generalized linear attention paradigm, which serves as
a low-rank approximation of a wide spectrum of popular linear token mixers. To
save the training cost and better leverage pre-trained models, we initialize
our models and distill the knowledge from pre-trained StableDiffusion (SD). We
find that the distilled model, termed LinFusion, achieves performance on par
with or superior to the original SD after only modest training, while
significantly reducing time and memory complexity. Extensive experiments on
SD-v1.5, SD-v2.1, and SD-XL demonstrate that LinFusion delivers satisfactory
zero-shot cross-resolution generation performance, generating high-resolution
images like 16K resolution. Moreover, it is highly compatible with pre-trained
SD components, such as ControlNet and IP-Adapter, requiring no adaptation
efforts. Codes are available at https://github.com/Huage001/LinFusion.
A novel linear attention mechanism, named LinFusion, is introduced to enhance high-resolution image generation with reduced computational cost, offering performance comparable to or better than existing diffusion models.
diffusion models Transformer-based UNet self-attention operations linear attention mechanism Mamba Mamba2 Gated Linear Attention attention normalization non-causal inference generalized linear attention paradigm low-rank approximation linear token mixers StableDiffusion (SD) zero-shot cross-resolution generation 16K resolution ControlNet IP-Adapter