Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language
  Models through Visual Counterfacts
Multimodal Large Language Models (MLLMs) perform well on tasks such as visual
question answering, but it remains unclear whether their reasoning relies more
on memorized world knowledge or on the visual information present in the input
image. To investigate this, we introduce Visual CounterFact, a new dataset of
visually-realistic counterfactuals that put world knowledge priors (e.g, red
strawberry) into direct conflict with visual input (e.g, blue strawberry).
Using Visual CounterFact, we show that model predictions initially reflect
memorized priors, but shift toward visual evidence in mid-to-late layers. This
dynamic reveals a competition between the two modalities, with visual input
ultimately overriding priors during evaluation. To control this behavior, we
propose Pixels Versus Priors (PvP) steering vectors, a mechanism for
controlling model outputs toward either world knowledge or visual input through
activation-level interventions. On average, PvP successfully shifts 92.5% of
color and 74.6% of size predictions from priors to counterfactuals. Together,
these findings offer new tools for interpreting and controlling factual
behavior in multimodal models.
683f4b081a2d176679fab869
Michal Golovanevsky
683f4b081a2d176679fab86a
William Rudman
683f4b081a2d176679fab86b
Michael Lepori
683f4b081a2d176679fab86c
Amir Bar
683f4b081a2d176679fab86d
Ritambhara Singh
683f4b081a2d176679fab86e
Carsten Eickhoff
Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language
  Models through Visual Counterfacts
Multimodal Large Language Models (MLLMs) perform well on tasks such as visual
question answering, but it remains unclear whether their reasoning relies more
on memorized world knowledge or on the visual information present in the input
image. To investigate this, we introduce Visual CounterFact, a new dataset of
visually-realistic counterfactuals that put world knowledge priors (e.g, red
strawberry) into direct conflict with visual input (e.g, blue strawberry).
Using Visual CounterFact, we show that model predictions initially reflect
memorized priors, but shift toward visual evidence in mid-to-late layers. This
dynamic reveals a competition between the two modalities, with visual input
ultimately overriding priors during evaluation. To control this behavior, we
propose Pixels Versus Priors (PvP) steering vectors, a mechanism for
controlling model outputs toward either world knowledge or visual input through
activation-level interventions. On average, PvP successfully shifts 92.5% of
color and 74.6% of size predictions from priors to counterfactuals. Together,
these findings offer new tools for interpreting and controlling factual
behavior in multimodal models.
Visual CounterFact and PvP steering vectors help interpret and control the competition between visual input and memorized world knowledge in multimodal large language models.
Multimodal Large Language Models Visual CounterFact visually-realistic counterfactuals world knowledge priors visual input model predictions mid-to-late layers Pixels Versus Priors (PvP) steering vectors activation-level interventions