Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
This work presents Switti, a scale-wise transformer for text-to-image
generation. Starting from existing next-scale prediction AR models, we first
explore them for T2I generation and propose architectural modifications to
improve their convergence and overall performance. We then observe that
self-attention maps of our pretrained scale-wise AR model exhibit weak
dependence on preceding scales. Based on this insight, we propose a non-AR
counterpart facilitating {sim}11% faster sampling and lower memory usage
while also achieving slightly better generation quality.Furthermore, we reveal
that classifier-free guidance at high-resolution scales is often unnecessary
and can even degrade performance. %may be not only unnecessary but potentially
detrimental. By disabling guidance at these scales, we achieve an additional
sampling acceleration of {sim}20% and improve the generation of
fine-grained details. Extensive human preference studies and automated
evaluations show that Switti outperforms existing T2I AR models and competes
with state-of-the-art T2I diffusion models while being up to 7{times}
faster.
674eb16e236b0a3fa92fdacd
Anton Voronov
admin_assigned
2024-12-03T13:48:25.474Z
6153161e0f96aa6c4d14cbf0
/avatars/e708193a493b850cf266ad6fb8a8beb7.svg
Anton Voronov
user
AntonVoronov
674eb16e236b0a3fa92fdace
Denis Kuznedelev
claimed_verified
2024-12-03T09:56:35.421Z
629cf0475a13ba8233dd18c9
https://cdn-avatars.huggingface.co/v1/production/uploads/1654452258405-noauth.jpeg
Denis Kuznedelev
user
SpiridonSunRotator
674eb16e236b0a3fa92fdacf
Mikhail Khoroshikh
admin_assigned
2024-12-03T13:48:31.974Z
628b8e83ac304a69264a8c9c
/avatars/a51b630e8d01224d5d8d5edff0a69d30.svg
Mikhail Khoroshikh
user
michellemoorre
674eb16e236b0a3fa92fdad0
Valentin Khrulkov
admin_assigned
2024-12-03T13:48:37.650Z
624c4c51be275a1baf50a19b
/avatars/aa527fff70d096955962e1964a6840c8.svg
Valentin Khrulkov
user
mathemage
674eb16e236b0a3fa92fdad1
Dmitry Baranchuk
admin_assigned
2024-12-03T13:48:43.760Z
62b6cc49752323892323bc04
https://cdn-avatars.huggingface.co/v1/production/uploads/62b6cc49752323892323bc04/gGBld1KJIP9AIpd81L3PC.jpeg
Dmitry Baranchuk
user
dbaranchuk
Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
This work presents Switti, a scale-wise transformer for text-to-image
generation. Starting from existing next-scale prediction AR models, we first
explore them for T2I generation and propose architectural modifications to
improve their convergence and overall performance. We then observe that
self-attention maps of our pretrained scale-wise AR model exhibit weak
dependence on preceding scales. Based on this insight, we propose a non-AR
counterpart facilitating {sim}11% faster sampling and lower memory usage
while also achieving slightly better generation quality.Furthermore, we reveal
that classifier-free guidance at high-resolution scales is often unnecessary
and can even degrade performance. %may be not only unnecessary but potentially
detrimental. By disabling guidance at these scales, we achieve an additional
sampling acceleration of {sim}20% and improve the generation of
fine-grained details. Extensive human preference studies and automated
evaluations show that Switti outperforms existing T2I AR models and competes
with state-of-the-art T2I diffusion models while being up to 7{times}
faster.
Switti, a scale-wise transformer for text-to-image generation, improves convergence and performance through architectural modifications, reduces memory usage, and achieves competitive results compared to diffusion models with significant speed advantages.
scale-wise transformer text-to-image generation next-scale prediction AR models self-attention maps pretrained scale-wise AR model non-AR counterpart classifier-free guidance high-resolution scales sampling acceleration human preference studies automated evaluations diffusion models