Precise Parameter Localization for Textual Generation in Diffusion
  Models
Novel diffusion models can synthesize photo-realistic images with integrated
high-quality text. Surprisingly, we demonstrate through attention activation
patching that only less than 1% of diffusion models' parameters, all contained
in attention layers, influence the generation of textual content within the
images. Building on this observation, we improve textual generation efficiency
and performance by targeting cross and joint attention layers of diffusion
models. We introduce several applications that benefit from localizing the
layers responsible for textual content generation. We first show that a
LoRA-based fine-tuning solely of the localized layers enhances, even more, the
general text-generation capabilities of large diffusion models while preserving
the quality and diversity of the diffusion models' generations. Then, we
demonstrate how we can use the localized layers to edit textual content in
generated images. Finally, we extend this idea to the practical use case of
preventing the generation of toxic text in a cost-free manner. In contrast to
prior work, our localization approach is broadly applicable across various
diffusion model architectures, including U-Net (e.g., LDM and SDXL) and
transformer-based (e.g., DeepFloyd IF and Stable Diffusion 3), utilizing
diverse text encoders (e.g., from CLIP to the large language models like T5).
Project page available at https://t2i-text-loc.github.io/.
67b2e6939edebc815a35eec8
Łukasz Staniszewski
claimed_verified
2025-02-18T09:32:40.751Z
63c7c19721bd95f80ed8ed80
/avatars/0b1c1ace991e0290118d4f99f619d809.svg
Lukasz Staniszewski
user
lukasz-staniszewski
67b2e6939edebc815a35eec9
Bartosz Cywiński
claimed_verified
2025-02-18T09:32:39.005Z
6422f416a73327caad9d1d86
/avatars/aa3639277cd1732504402fc64a57eff8.svg
Bartosz Cywiński
user
bcywinski
67b2e6939edebc815a35eeca
Franziska Boenisch
67b2e6939edebc815a35eecb
Kamil Deja
67b2e6939edebc815a35eecc
Adam Dziedzic
Precise Parameter Localization for Textual Generation in Diffusion
  Models
Novel diffusion models can synthesize photo-realistic images with integrated
high-quality text. Surprisingly, we demonstrate through attention activation
patching that only less than 1% of diffusion models' parameters, all contained
in attention layers, influence the generation of textual content within the
images. Building on this observation, we improve textual generation efficiency
and performance by targeting cross and joint attention layers of diffusion
models. We introduce several applications that benefit from localizing the
layers responsible for textual content generation. We first show that a
LoRA-based fine-tuning solely of the localized layers enhances, even more, the
general text-generation capabilities of large diffusion models while preserving
the quality and diversity of the diffusion models' generations. Then, we
demonstrate how we can use the localized layers to edit textual content in
generated images. Finally, we extend this idea to the practical use case of
preventing the generation of toxic text in a cost-free manner. In contrast to
prior work, our localization approach is broadly applicable across various
diffusion model architectures, including U-Net (e.g., LDM and SDXL) and
transformer-based (e.g., DeepFloyd IF and Stable Diffusion 3), utilizing
diverse text encoders (e.g., from CLIP to the large language models like T5).
Project page available at https://t2i-text-loc.github.io/.
By localizing attention layers in diffusion models, text generation efficiency and performance improve, enabling text editing, toxic text prevention, and enhanced generalization across various architectures.
diffusion models photo-realistic images attention layers LoRA-based fine-tuning U-Net DeepFloyd IF Stable Diffusion 3 CLIP T5