MLLMs Know Where to Look: Training-free Perception of Small Visual
  Details with Multimodal LLMs
Multimodal Large Language Models (MLLMs) have experienced rapid progress in
visual recognition tasks in recent years. Given their potential integration
into many critical applications, it is important to understand the limitations
of their visual perception. In this work, we study whether MLLMs can perceive
small visual details as effectively as large ones when answering questions
about images. We observe that their performance is very sensitive to the size
of the visual subject of the question, and further show that this effect is in
fact causal by conducting an intervention study. Next, we study the attention
patterns of MLLMs when answering visual questions, and intriguingly find that
they consistently know where to look, even when they provide the wrong answer.
Based on these findings, we then propose training-free visual intervention
methods that leverage the internal knowledge of any MLLM itself, in the form of
attention and gradient maps, to enhance its perception of small visual details.
We evaluate our proposed methods on two widely-used MLLMs and seven visual
question answering benchmarks and show that they can significantly improve
MLLMs' accuracy without requiring any training. Our results elucidate the risk
of applying MLLMs to visual recognition tasks concerning small details and
indicate that visual intervention using the model's internal state is a
promising direction to mitigate this risk.
67bf6ea633d6740f711cc995
Jiarui Zhang
claimed_verified
2025-02-27T09:15:01.691Z
635b99d47a1656011516bff9
/avatars/7243c4171ff127ba90631f105881d9d7.svg
jiarui zhang
user
jrzhang
67bf6ea633d6740f711cc996
Mahyar Khayatkhoei
67bf6ea633d6740f711cc997
Prateek Chhikara
claimed_verified
2025-04-30T07:57:10.647Z
63d81cb4635499f2164094bf
https://cdn-avatars.huggingface.co/v1/production/uploads/1675107437601-noauth.png
Prateek Chhikara
user
prateekchhikara
67bf6ea633d6740f711cc998
Filip Ilievski
MLLMs Know Where to Look: Training-free Perception of Small Visual
  Details with Multimodal LLMs
Multimodal Large Language Models (MLLMs) have experienced rapid progress in
visual recognition tasks in recent years. Given their potential integration
into many critical applications, it is important to understand the limitations
of their visual perception. In this work, we study whether MLLMs can perceive
small visual details as effectively as large ones when answering questions
about images. We observe that their performance is very sensitive to the size
of the visual subject of the question, and further show that this effect is in
fact causal by conducting an intervention study. Next, we study the attention
patterns of MLLMs when answering visual questions, and intriguingly find that
they consistently know where to look, even when they provide the wrong answer.
Based on these findings, we then propose training-free visual intervention
methods that leverage the internal knowledge of any MLLM itself, in the form of
attention and gradient maps, to enhance its perception of small visual details.
We evaluate our proposed methods on two widely-used MLLMs and seven visual
question answering benchmarks and show that they can significantly improve
MLLMs' accuracy without requiring any training. Our results elucidate the risk
of applying MLLMs to visual recognition tasks concerning small details and
indicate that visual intervention using the model's internal state is a
promising direction to mitigate this risk.
MLLMs perform poorly on visual details of small size, but attention and gradient map interventions can enhance their accuracy without training.
Multimodal Large Language Models MLLMs visual recognition visual perception visual questions attention patterns gradient maps visual intervention