Textual Steering Vectors Can Improve Visual Understanding in Multimodal
  Large Language Models
Steering methods have emerged as effective and targeted tools for guiding
large language models' (LLMs) behavior without modifying their parameters.
Multimodal large language models (MLLMs), however, do not currently enjoy the
same suite of techniques, due in part to their recency and architectural
diversity. Inspired by this gap, we investigate whether MLLMs can be steered
using vectors derived from their text-only LLM backbone, via sparse
autoencoders (SAEs), mean shift, and linear probing. We find that text-derived
steering consistently enhances multimodal accuracy across diverse MLLM
architectures and visual tasks. In particular, mean shift boosts spatial
relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to
+3.3%, outperforming prompting and exhibiting strong generalization to
out-of-distribution datasets. These results highlight textual steering vectors
as a powerful, efficient mechanism for enhancing grounding in MLLMs with
minimal additional data collection and computational overhead.
6836384a1ec776c1b0fc0d7f
Woody Haosheng Gan
6836384a1ec776c1b0fc0d80
Deqing Fu
claimed_verified
2025-05-28T08:58:38.089Z
63c8454e46421a2efe82709d
https://cdn-avatars.huggingface.co/v1/production/uploads/63c8454e46421a2efe82709d/3BcSk4KOwAgWHEPVtsAV3.png
Deqing Fu
user
deqing
6836384a1ec776c1b0fc0d81
Julian Asilis
6836384a1ec776c1b0fc0d82
Ollie Liu
6836384a1ec776c1b0fc0d83
Dani Yogatama
6836384a1ec776c1b0fc0d84
Vatsal Sharan
6836384a1ec776c1b0fc0d85
Robin Jia
6836384a1ec776c1b0fc0d86
Willie Neiswanger
claimed_verified
2025-06-13T07:42:23.062Z
644bf65522d211df6444a7f4
https://cdn-avatars.huggingface.co/v1/production/uploads/644bf65522d211df6444a7f4/k_ddZdQDg2fzhwjI1EXyx.jpeg
Willie Neiswanger
user
willieneis
Textual Steering Vectors Can Improve Visual Understanding in Multimodal
  Large Language Models
Steering methods have emerged as effective and targeted tools for guiding
large language models' (LLMs) behavior without modifying their parameters.
Multimodal large language models (MLLMs), however, do not currently enjoy the
same suite of techniques, due in part to their recency and architectural
diversity. Inspired by this gap, we investigate whether MLLMs can be steered
using vectors derived from their text-only LLM backbone, via sparse
autoencoders (SAEs), mean shift, and linear probing. We find that text-derived
steering consistently enhances multimodal accuracy across diverse MLLM
architectures and visual tasks. In particular, mean shift boosts spatial
relationship accuracy on CV-Bench by up to +7.3% and counting accuracy by up to
+3.3%, outperforming prompting and exhibiting strong generalization to
out-of-distribution datasets. These results highlight textual steering vectors
as a powerful, efficient mechanism for enhancing grounding in MLLMs with
minimal additional data collection and computational overhead.
Text-derived steering via sparse autoencoders, mean shift, and linear probing enhances multimodal accuracy in large language models without requiring parameter modifications or significant additional data or computation.
sparse autoencoders mean shift linear probing multimodal large language models CV-Bench spatial relationship accuracy counting accuracy textual steering vectors grounding