SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
  Teacher
In this paper, we aim to enhance the performance of SwiftBrush, a prominent
one-step text-to-image diffusion model, to be competitive with its multi-step
Stable Diffusion counterpart. Initially, we explore the quality-diversity
trade-off between SwiftBrush and SD Turbo: the former excels in image
diversity, while the latter excels in image quality. This observation motivates
our proposed modifications in the training methodology, including better weight
initialization and efficient LoRA training. Moreover, our introduction of a
novel clamped CLIP loss enhances image-text alignment and results in improved
image quality. Remarkably, by combining the weights of models trained with
efficient LoRA and full training, we achieve a new state-of-the-art one-step
diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
multi-step Stable Diffusion models. The evaluation code is available at:
https://github.com/vinairesearch/swiftbrushv2.
66cd4bff332ce5c746878e46
Trung Dao
admin_assigned
2024-08-27T10:30:48.275Z
64d265019badd06a0587c848
https://cdn-avatars.huggingface.co/v1/production/uploads/64d265019badd06a0587c848/qrOtJ7gxUCzgVwduFDktf.jpeg
Dao Tuan Trung
user
termanteus
66cd4bff332ce5c746878e47
Thuan Hoang Nguyen
claimed_verified
2024-08-27T07:22:45.751Z
633d4b6bb8ac3a16a5181ec2
https://cdn-avatars.huggingface.co/v1/production/uploads/633d4b6bb8ac3a16a5181ec2/bgD_lF2NwREkWIB4yki4q.jpeg
Thuan Nguyen Hoang
user
thuanz123
66cd4bff332ce5c746878e48
Thanh Le
claimed_verified
2024-08-27T10:42:37.270Z
603472929aa44495c80dd049
https://cdn-avatars.huggingface.co/v1/production/uploads/1619792526143-603472929aa44495c80dd049.jpeg
Thành Lê
user
imthanhlv
66cd4bff332ce5c746878e49
Duc Vu
claimed_verified
2024-08-27T10:31:08.519Z
636b9364ecd380d10a8f4a83
/avatars/ae390369c04dca24125ab05f3f0da250.svg
Duc Vu
user
ducvh5
66cd4bff332ce5c746878e4a
Khoi Nguyen
claimed_verified
2024-08-27T07:22:44.073Z
63044075d14428368d1c4c6c
https://cdn-avatars.huggingface.co/v1/production/uploads/63044075d14428368d1c4c6c/P2WhQMib5VXyT0w-Sjf2V.jpeg
Khoi Nguyen
user
ducminhkhoi
66cd4bff332ce5c746878e4b
Cuong Pham
66cd4bff332ce5c746878e4c
Anh Tran
admin_assigned
2024-08-27T08:15:20.425Z
64b3eb404c3cc95a7562563d
/avatars/5313d53807663e3b874391ec3f1a974a.svg
ANH TRAN
user
anhtran
SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
  Teacher
In this paper, we aim to enhance the performance of SwiftBrush, a prominent
one-step text-to-image diffusion model, to be competitive with its multi-step
Stable Diffusion counterpart. Initially, we explore the quality-diversity
trade-off between SwiftBrush and SD Turbo: the former excels in image
diversity, while the latter excels in image quality. This observation motivates
our proposed modifications in the training methodology, including better weight
initialization and efficient LoRA training. Moreover, our introduction of a
novel clamped CLIP loss enhances image-text alignment and results in improved
image quality. Remarkably, by combining the weights of models trained with
efficient LoRA and full training, we achieve a new state-of-the-art one-step
diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
multi-step Stable Diffusion models. The evaluation code is available at:
https://github.com/vinairesearch/swiftbrushv2.
Enhancements to SwiftBrush through improved weight initialization, efficient LoRA training, and novel clamped CLIP loss achieve state-of-the-art image quality while maintaining generative diversity.
one-step text-to-image diffusion model multi-step Stable Diffusion quality-diversity trade-off weight initialization efficient LoRA training clamped CLIP loss image-text alignment FID