title: Sparse Autoencoders Learn Monosemantic Features in Vision-Language
    Models
summary: Sparse Autoencoders (SAEs) have recently been shown to enhance
  interpretability and steerability in Large Language Models (LLMs). In this
  work, we extend the application of SAEs to Vision-Language Models (VLMs), such
  as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
  in vision representations. Our experimental results reveal that SAEs trained on
  VLMs significantly enhance the monosemanticity of individual neurons while also
  exhibiting hierarchical representations that align well with expert-defined
  structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
  applying SAEs to intervene on a CLIP vision encoder, directly steer output from
  multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
  model. These findings emphasize the practicality and efficacy of SAEs as an
  unsupervised approach for enhancing both the interpretability and control of
  VLMs.
paper_authors._id: 67ef9a8885ea9d1d7db3fdb5
paper_authors.name: Mateusz Pach
paper_authors._id: 67ef9a8885ea9d1d7db3fdb6
paper_authors.name: Shyamgopal Karthik
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-08T06:54:15.309Z
paper_authors._id: 6254599b6e36fe62e141c8f9
paper_authors.avatarUrl: /avatars/08d6b68b92c7bfd0a8be022ba9f2f289.svg
paper_authors.fullname: Shyamgopal Karthik
paper_authors.type: user
paper_authors.user: shyamgopal
paper_authors._id: 67ef9a8885ea9d1d7db3fdb7
paper_authors.name: Quentin Bouniot
paper_authors._id: 67ef9a8885ea9d1d7db3fdb8
paper_authors.name: Serge Belongie
paper_authors._id: 67ef9a8885ea9d1d7db3fdb9
paper_authors.name: Zeynep Akata
paper_title: Sparse Autoencoders Learn Monosemantic Features in Vision-Language
    Models
paper_summary: Sparse Autoencoders (SAEs) have recently been shown to enhance
  interpretability and steerability in Large Language Models (LLMs). In this
  work, we extend the application of SAEs to Vision-Language Models (VLMs), such
  as CLIP, and introduce a comprehensive framework for evaluating monosemanticity
  in vision representations. Our experimental results reveal that SAEs trained on
  VLMs significantly enhance the monosemanticity of individual neurons while also
  exhibiting hierarchical representations that align well with expert-defined
  structures (e.g., iNaturalist taxonomy). Most notably, we demonstrate that
  applying SAEs to intervene on a CLIP vision encoder, directly steer output from
  multimodal LLMs (e.g., LLaVA) without any modifications to the underlying
  model. These findings emphasize the practicality and efficacy of SAEs as an
  unsupervised approach for enhancing both the interpretability and control of
  VLMs.
paper_ai_summary: Sparse Autoencoders (SAEs) enhance monosemanticity and hierarchical representations in Vision-Language Models (VLMs) such as CLIP, allowing intervention without modifying the underlying model.
paper_ai_keywords: Sparse Autoencoders SAEs Vision-Language Models VLMs CLIP multimodal LLMs interpretability monosemanticity hierarchical representations iNaturalist taxonomy LLaVA