title: Personalized Residuals for Concept-Driven Text-to-Image Generation
summary: We present personalized residuals and localized attention-guided sampling for
  efficient concept-driven generation using text-to-image diffusion models. Our
  method first represents concepts by freezing the weights of a pretrained
  text-conditioned diffusion model and learning low-rank residuals for a small
  subset of the model's layers. The residual-based approach then directly enables
  application of our proposed sampling technique, which applies the learned
  residuals only in areas where the concept is localized via cross-attention and
  applies the original diffusion weights in all other regions. Localized sampling
  therefore combines the learned identity of the concept with the existing
  generative prior of the underlying diffusion model. We show that personalized
  residuals effectively capture the identity of a concept in ~3 minutes on a
  single GPU without the use of regularization images and with fewer parameters
  than previous models, and localized sampling allows using the original model as
  strong prior for large parts of the image.
paper_authors._id: 664d5a41220a26fca69acf4b
paper_authors.name: Cusuh Ham
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-22T13:29:30.428Z
paper_authors._id: 6372fab1bd1595ae66a62543
paper_authors.avatarUrl: /avatars/783bdae07b2663eebeea4c7919a87c91.svg
paper_authors.fullname: Cusuh Ham
paper_authors.type: user
paper_authors.user: cusuh
paper_authors._id: 664d5a41220a26fca69acf4c
paper_authors.name: Matthew Fisher
paper_authors._id: 664d5a41220a26fca69acf4d
paper_authors.name: James Hays
paper_authors._id: 664d5a41220a26fca69acf4e
paper_authors.name: Nicholas Kolkin
paper_authors._id: 664d5a41220a26fca69acf4f
paper_authors.name: Yuchen Liu
paper_authors._id: 664d5a41220a26fca69acf50
paper_authors.name: Richard Zhang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-05-22T02:36:53.222Z
paper_authors._id: 6308f7ed3485f37782a32a8c
paper_authors.avatarUrl: /avatars/cc129bef87850384941a6f6f293c08d9.svg
paper_authors.fullname: Richard Zhang
paper_authors.type: user
paper_authors.user: rzhang
paper_authors._id: 664d5a41220a26fca69acf51
paper_authors.name: Tobias Hinz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-22T09:04:00.017Z
paper_authors._id: 65d4aa23b16cabdb982bb309
paper_authors.avatarUrl: /avatars/9d5f17cd06ec7620a0261610540ea534.svg
paper_authors.fullname: Tobias Hinz
paper_authors.type: user
paper_authors.user: phazerphoo
paper_title: Personalized Residuals for Concept-Driven Text-to-Image Generation
paper_summary: We present personalized residuals and localized attention-guided sampling for
  efficient concept-driven generation using text-to-image diffusion models. Our
  method first represents concepts by freezing the weights of a pretrained
  text-conditioned diffusion model and learning low-rank residuals for a small
  subset of the model's layers. The residual-based approach then directly enables
  application of our proposed sampling technique, which applies the learned
  residuals only in areas where the concept is localized via cross-attention and
  applies the original diffusion weights in all other regions. Localized sampling
  therefore combines the learned identity of the concept with the existing
  generative prior of the underlying diffusion model. We show that personalized
  residuals effectively capture the identity of a concept in ~3 minutes on a
  single GPU without the use of regularization images and with fewer parameters
  than previous models, and localized sampling allows using the original model as
  strong prior for large parts of the image.
paper_ai_keywords: personalized tuning attention-guided sampling concept-driven generation text-to-image diffusion models pretrained text-conditioned diffusion model low-rank residuals cross-attention localized sampling generative prior personalized residuals parameter-efficient fine-tuning