title: Textual Steering Vectors Can Improve Visual Understanding in Multimodal
    Large Language Models
summary: Steering methods have emerged as effective and targeted tools for guiding
  large language models' (LLMs) behavior without modifying their parameters.
  Multimodal large language models (MLLMs), however, do not currently enjoy the
  same suite of techniques, due in part to their recency and architectural
  diversity. Inspired by this gap, we investigate whether MLLMs can be steered
  using vectors derived from their text-only LLM backbone, via sparse
  autoencoders (SAEs), mean shift, and linear probing. We find that text-derived
  steering consistently enhances multimodal accuracy across diverse MLLM
  architectures and visual tasks. In particular, mean shift boosts spatial
  relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to
  +3.3%, outperforming prompting and exhibiting strong generalization to
  out-of-distribution datasets. These results highlight textual steering vectors
  as a powerful, efficient mechanism for enhancing grounding in MLLMs with
  minimal additional data collection and computational overhead.
paper_authors._id: 6836384a1ec776c1b0fc0d7f
paper_authors.name: Woody Haosheng Gan
paper_authors._id: 6836384a1ec776c1b0fc0d80
paper_authors.name: Deqing Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T08:58:38.089Z
paper_authors._id: 63c8454e46421a2efe82709d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
paper_authors.fullname: Deqing Fu
paper_authors.type: user
paper_authors.user: deqing
paper_authors._id: 6836384a1ec776c1b0fc0d81
paper_authors.name: Julian Asilis
paper_authors._id: 6836384a1ec776c1b0fc0d82
paper_authors.name: Ollie Liu
paper_authors._id: 6836384a1ec776c1b0fc0d83
paper_authors.name: Dani Yogatama
paper_authors._id: 6836384a1ec776c1b0fc0d84
paper_authors.name: Vatsal Sharan
paper_authors._id: 6836384a1ec776c1b0fc0d85
paper_authors.name: Robin Jia
paper_authors._id: 6836384a1ec776c1b0fc0d86
paper_authors.name: Willie Neiswanger
paper_title: Textual Steering Vectors Can Improve Visual Understanding in Multimodal
    Large Language Models
paper_summary: Steering methods have emerged as effective and targeted tools for guiding
  large language models' (LLMs) behavior without modifying their parameters.
  Multimodal large language models (MLLMs), however, do not currently enjoy the
  same suite of techniques, due in part to their recency and architectural
  diversity. Inspired by this gap, we investigate whether MLLMs can be steered
  using vectors derived from their text-only LLM backbone, via sparse
  autoencoders (SAEs), mean shift, and linear probing. We find that text-derived
  steering consistently enhances multimodal accuracy across diverse MLLM
  architectures and visual tasks. In particular, mean shift boosts spatial
  relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to
  +3.3%, outperforming prompting and exhibiting strong generalization to
  out-of-distribution datasets. These results highlight textual steering vectors
  as a powerful, efficient mechanism for enhancing grounding in MLLMs with
  minimal additional data collection and computational overhead.
paper_ai_summary: Text-derived steering via sparse autoencoders, mean shift, and linear probing enhances multimodal accuracy in large language models without requiring parameter modifications or significant additional data or computation.
paper_ai_keywords: sparse autoencoders mean shift linear probing multimodal large language models CV-Bench spatial relationship accuracy counting accuracy textual steering vectors grounding