Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable
  Generations
Despite the remarkable progress in text-to-image generative models, they are
prone to adversarial attacks and inadvertently generate unsafe, unethical
content. Existing approaches often rely on fine-tuning models to remove
specific concepts, which is computationally expensive, lack scalability, and/or
compromise generation quality. In this work, we propose a novel framework
leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable
concept manipulation in diffusion models. Specifically, we first identify
interpretable monosemantic concepts in the latent space of text embeddings and
leverage them to precisely steer the generation away or towards a given concept
(e.g., nudity) or to introduce a new concept (e.g., photographic style).
Through extensive experiments, we demonstrate that our approach is very simple,
requires no retraining of the base model nor LoRA adapters, does not compromise
the generation quality, and is robust to adversarial prompt manipulations. Our
method yields an improvement of 20.01% in unsafe concept removal,
is effective in style manipulation, and is sim5x faster than
current state-of-the-art.
67a0f59c5685d37e28880943
Dahye Kim
claimed_verified
2025-02-03T16:59:53.099Z
64bcc06fb567ae97c3272d3d
/avatars/bcb61fe9e575154d84913a1501971f1a.svg
kim
user
dahyekim
67a0f59c5685d37e28880944
Deepti Ghadiyaram
Concept Steerers: Leveraging K-Sparse Autoencoders for Controllable
  Generations
Despite the remarkable progress in text-to-image generative models, they are
prone to adversarial attacks and inadvertently generate unsafe, unethical
content. Existing approaches often rely on fine-tuning models to remove
specific concepts, which is computationally expensive, lack scalability, and/or
compromise generation quality. In this work, we propose a novel framework
leveraging k-sparse autoencoders (k-SAEs) to enable efficient and interpretable
concept manipulation in diffusion models. Specifically, we first identify
interpretable monosemantic concepts in the latent space of text embeddings and
leverage them to precisely steer the generation away or towards a given concept
(e.g., nudity) or to introduce a new concept (e.g., photographic style).
Through extensive experiments, we demonstrate that our approach is very simple,
requires no retraining of the base model nor LoRA adapters, does not compromise
the generation quality, and is robust to adversarial prompt manipulations. Our
method yields an improvement of 20.01% in unsafe concept removal,
is effective in style manipulation, and is sim5x faster than
current state-of-the-art.
The framework using k-sparse autoencoders in diffusion models enables efficient and interpretable concept manipulation, improving unsafe concept removal and style manipulation without retraining.
k-sparse autoencoders k-SAEs diffusion models latent space text embeddings monosemantic concepts prompt manipulations