Personalized Residuals for Concept-Driven Text-to-Image Generation
We present personalized residuals and localized attention-guided sampling for
efficient concept-driven generation using text-to-image diffusion models. Our
method first represents concepts by freezing the weights of a pretrained
text-conditioned diffusion model and learning low-rank residuals for a small
subset of the model's layers. The residual-based approach then directly enables
application of our proposed sampling technique, which applies the learned
residuals only in areas where the concept is localized via cross-attention and
applies the original diffusion weights in all other regions. Localized sampling
therefore combines the learned identity of the concept with the existing
generative prior of the underlying diffusion model. We show that personalized
residuals effectively capture the identity of a concept in ~3 minutes on a
single GPU without the use of regularization images and with fewer parameters
than previous models, and localized sampling allows using the original model as
strong prior for large parts of the image.
664d5a41220a26fca69acf4b
Cusuh Ham
extracted_confirmed
2024-05-22T13:29:30.428Z
6372fab1bd1595ae66a62543
/avatars/783bdae07b2663eebeea4c7919a87c91.svg
Cusuh Ham
user
cusuh
664d5a41220a26fca69acf4c
Matthew Fisher
664d5a41220a26fca69acf4d
James Hays
664d5a41220a26fca69acf4e
Nicholas Kolkin
664d5a41220a26fca69acf4f
Yuchen Liu
664d5a41220a26fca69acf50
Richard Zhang
extracted_pending
2024-05-22T02:36:53.222Z
6308f7ed3485f37782a32a8c
/avatars/cc129bef87850384941a6f6f293c08d9.svg
Richard Zhang
user
rzhang
664d5a41220a26fca69acf51
Tobias Hinz
admin_assigned
2024-05-22T09:04:00.017Z
65d4aa23b16cabdb982bb309
/avatars/9d5f17cd06ec7620a0261610540ea534.svg
Tobias Hinz
user
phazerphoo
Personalized Residuals for Concept-Driven Text-to-Image Generation
We present personalized residuals and localized attention-guided sampling for
efficient concept-driven generation using text-to-image diffusion models. Our
method first represents concepts by freezing the weights of a pretrained
text-conditioned diffusion model and learning low-rank residuals for a small
subset of the model's layers. The residual-based approach then directly enables
application of our proposed sampling technique, which applies the learned
residuals only in areas where the concept is localized via cross-attention and
applies the original diffusion weights in all other regions. Localized sampling
therefore combines the learned identity of the concept with the existing
generative prior of the underlying diffusion model. We show that personalized
residuals effectively capture the identity of a concept in ~3 minutes on a
single GPU without the use of regularization images and with fewer parameters
than previous models, and localized sampling allows using the original model as
strong prior for large parts of the image.
Personalized residuals and localized attention-guided sampling enhance concept-driven text-to-image generation by efficiently capturing concept identity and leveraging existing diffusion model priors.
personalized residuals localized attention text-to-image diffusion models concept-driven generation low-rank residuals cross-attention localized sampling generative prior