title: Precise Parameter Localization for Textual Generation in Diffusion
    Models
summary: Novel diffusion models can synthesize photo-realistic images with integrated
  high-quality text. Surprisingly, we demonstrate through attention activation
  patching that only less than 1% of diffusion models' parameters, all contained
  in attention layers, influence the generation of textual content within the
  images. Building on this observation, we improve textual generation efficiency
  and performance by targeting cross and joint attention layers of diffusion
  models. We introduce several applications that benefit from localizing the
  layers responsible for textual content generation. We first show that a
  LoRA-based fine-tuning solely of the localized layers enhances, even more, the
  general text-generation capabilities of large diffusion models while preserving
  the quality and diversity of the diffusion models' generations. Then, we
  demonstrate how we can use the localized layers to edit textual content in
  generated images. Finally, we extend this idea to the practical use case of
  preventing the generation of toxic text in a cost-free manner. In contrast to
  prior work, our localization approach is broadly applicable across various
  diffusion model architectures, including U-Net (e.g., LDM and SDXL) and
  transformer-based (e.g., DeepFloyd IF and Stable Diffusion 3), utilizing
  diverse text encoders (e.g., from CLIP to the large language models like T5).
  Project page available at https://t2i-text-loc.github.io/.
paper_authors._id: 67b2e6939edebc815a35eec8
paper_authors.name: Łukasz Staniszewski
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:40.751Z
paper_authors._id: 63c7c19721bd95f80ed8ed80
paper_authors.avatarUrl: /avatars/0b1c1ace991e0290118d4f99f619d809.svg
paper_authors.fullname: Lukasz Staniszewski
paper_authors.type: user
paper_authors.user: lukasz-staniszewski
paper_authors._id: 67b2e6939edebc815a35eec9
paper_authors.name: Bartosz Cywiński
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:32:39.005Z
paper_authors._id: 6422f416a73327caad9d1d86
paper_authors.avatarUrl: /avatars/aa3639277cd1732504402fc64a57eff8.svg
paper_authors.fullname: Bartosz Cywiński
paper_authors.type: user
paper_authors.user: bcywinski
paper_authors._id: 67b2e6939edebc815a35eeca
paper_authors.name: Franziska Boenisch
paper_authors._id: 67b2e6939edebc815a35eecb
paper_authors.name: Kamil Deja
paper_authors._id: 67b2e6939edebc815a35eecc
paper_authors.name: Adam Dziedzic
paper_title: Precise Parameter Localization for Textual Generation in Diffusion
    Models
paper_summary: Novel diffusion models can synthesize photo-realistic images with integrated
  high-quality text. Surprisingly, we demonstrate through attention activation
  patching that only less than 1% of diffusion models' parameters, all contained
  in attention layers, influence the generation of textual content within the
  images. Building on this observation, we improve textual generation efficiency
  and performance by targeting cross and joint attention layers of diffusion
  models. We introduce several applications that benefit from localizing the
  layers responsible for textual content generation. We first show that a
  LoRA-based fine-tuning solely of the localized layers enhances, even more, the
  general text-generation capabilities of large diffusion models while preserving
  the quality and diversity of the diffusion models' generations. Then, we
  demonstrate how we can use the localized layers to edit textual content in
  generated images. Finally, we extend this idea to the practical use case of
  preventing the generation of toxic text in a cost-free manner. In contrast to
  prior work, our localization approach is broadly applicable across various
  diffusion model architectures, including U-Net (e.g., LDM and SDXL) and
  transformer-based (e.g., DeepFloyd IF and Stable Diffusion 3), utilizing
  diverse text encoders (e.g., from CLIP to the large language models like T5).
  Project page available at https://t2i-text-loc.github.io/.
paper_ai_summary: By localizing attention layers in diffusion models, text generation efficiency and performance improve, enabling text editing, toxic text prevention, and enhanced generalization across various architectures.
paper_ai_keywords: diffusion models photo-realistic images attention layers LoRA-based fine-tuning U-Net DeepFloyd IF Stable Diffusion 3 CLIP T5