title: SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
    Teacher
summary: In this paper, we aim to enhance the performance of SwiftBrush, a prominent
  one-step text-to-image diffusion model, to be competitive with its multi-step
  Stable Diffusion counterpart. Initially, we explore the quality-diversity
  trade-off between SwiftBrush and SD Turbo: the former excels in image
  diversity, while the latter excels in image quality. This observation motivates
  our proposed modifications in the training methodology, including better weight
  initialization and efficient LoRA training. Moreover, our introduction of a
  novel clamped CLIP loss enhances image-text alignment and results in improved
  image quality. Remarkably, by combining the weights of models trained with
  efficient LoRA and full training, we achieve a new state-of-the-art one-step
  diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
  multi-step Stable Diffusion models. The evaluation code is available at:
  https://github.com/vinairesearch/swiftbrushv2.
paper_authors._id: 66cd4bff332ce5c746878e46
paper_authors.name: Trung Dao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T10:30:48.275Z
paper_authors._id: 64d265019badd06a0587c848
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64d265019badd06a0587c848/qrOtJ7gxUCzgVwduFDktf.jpeg
paper_authors.fullname: Dao Tuan Trung
paper_authors.type: user
paper_authors.user: termanteus
paper_authors._id: 66cd4bff332ce5c746878e47
paper_authors.name: Thuan Hoang Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:22:45.751Z
paper_authors._id: 633d4b6bb8ac3a16a5181ec2
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/633d4b6bb8ac3a16a5181ec2/bgD_lF2NwREkWIB4yki4q.jpeg
paper_authors.fullname: Thuan Nguyen Hoang
paper_authors.type: user
paper_authors.user: thuanz123
paper_authors._id: 66cd4bff332ce5c746878e48
paper_authors.name: Thanh Le
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T10:42:37.270Z
paper_authors._id: 603472929aa44495c80dd049
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619792526143-603472929aa44495c80dd049.jpeg
paper_authors.fullname: Thành Lê
paper_authors.type: user
paper_authors.user: imthanhlv
paper_authors._id: 66cd4bff332ce5c746878e49
paper_authors.name: Duc Vu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T10:31:08.519Z
paper_authors._id: 636b9364ecd380d10a8f4a83
paper_authors.avatarUrl: /avatars/ae390369c04dca24125ab05f3f0da250.svg
paper_authors.fullname: Duc Vu
paper_authors.type: user
paper_authors.user: ducvh5
paper_authors._id: 66cd4bff332ce5c746878e4a
paper_authors.name: Khoi Nguyen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T07:22:44.073Z
paper_authors._id: 63044075d14428368d1c4c6c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63044075d14428368d1c4c6c/P2WhQMib5VXyT0w-Sjf2V.jpeg
paper_authors.fullname: Khoi Nguyen
paper_authors.type: user
paper_authors.user: ducminhkhoi
paper_authors._id: 66cd4bff332ce5c746878e4b
paper_authors.name: Cuong Pham
paper_authors._id: 66cd4bff332ce5c746878e4c
paper_authors.name: Anh Tran
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-27T08:15:20.425Z
paper_authors._id: 64b3eb404c3cc95a7562563d
paper_authors.avatarUrl: /avatars/5313d53807663e3b874391ec3f1a974a.svg
paper_authors.fullname: ANH TRAN
paper_authors.type: user
paper_authors.user: anhtran
paper_title: SwiftBrush v2: Make Your One-step Diffusion Model Better Than Its
    Teacher
paper_summary: In this paper, we aim to enhance the performance of SwiftBrush, a prominent
  one-step text-to-image diffusion model, to be competitive with its multi-step
  Stable Diffusion counterpart. Initially, we explore the quality-diversity
  trade-off between SwiftBrush and SD Turbo: the former excels in image
  diversity, while the latter excels in image quality. This observation motivates
  our proposed modifications in the training methodology, including better weight
  initialization and efficient LoRA training. Moreover, our introduction of a
  novel clamped CLIP loss enhances image-text alignment and results in improved
  image quality. Remarkably, by combining the weights of models trained with
  efficient LoRA and full training, we achieve a new state-of-the-art one-step
  diffusion model, achieving an FID of 8.14 and surpassing all GAN-based and
  multi-step Stable Diffusion models. The evaluation code is available at:
  https://github.com/vinairesearch/swiftbrushv2.
paper_ai_summary: Enhancements to SwiftBrush through improved weight initialization, efficient LoRA training, and novel clamped CLIP loss achieve state-of-the-art image quality while maintaining generative diversity.
paper_ai_keywords: one-step text-to-image diffusion model multi-step Stable Diffusion quality-diversity trade-off weight initialization efficient LoRA training clamped CLIP loss image-text alignment FID