title: MLLMs Know Where to Look: Training-free Perception of Small Visual
    Details with Multimodal LLMs
summary: Multimodal Large Language Models (MLLMs) have experienced rapid progress in
  visual recognition tasks in recent years. Given their potential integration
  into many critical applications, it is important to understand the limitations
  of their visual perception. In this work, we study whether MLLMs can perceive
  small visual details as effectively as large ones when answering questions
  about images. We observe that their performance is very sensitive to the size
  of the visual subject of the question, and further show that this effect is in
  fact causal by conducting an intervention study. Next, we study the attention
  patterns of MLLMs when answering visual questions, and intriguingly find that
  they consistently know where to look, even when they provide the wrong answer.
  Based on these findings, we then propose training-free visual intervention
  methods that leverage the internal knowledge of any MLLM itself, in the form of
  attention and gradient maps, to enhance its perception of small visual details.
  We evaluate our proposed methods on two widely-used MLLMs and seven visual
  question answering benchmarks and show that they can significantly improve
  MLLMs' accuracy without requiring any training. Our results elucidate the risk
  of applying MLLMs to visual recognition tasks concerning small details and
  indicate that visual intervention using the model's internal state is a
  promising direction to mitigate this risk.
paper_authors._id: 67bf6ea633d6740f711cc995
paper_authors.name: Jiarui Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-27T09:15:01.691Z
paper_authors._id: 635b99d47a1656011516bff9
paper_authors.avatarUrl: /avatars/7243c4171ff127ba90631f105881d9d7.svg
paper_authors.fullname: jiarui zhang
paper_authors.type: user
paper_authors.user: jrzhang
paper_authors._id: 67bf6ea633d6740f711cc996
paper_authors.name: Mahyar Khayatkhoei
paper_authors._id: 67bf6ea633d6740f711cc997
paper_authors.name: Prateek Chhikara
paper_authors._id: 67bf6ea633d6740f711cc998
paper_authors.name: Filip Ilievski
paper_title: MLLMs Know Where to Look: Training-free Perception of Small Visual
    Details with Multimodal LLMs
paper_summary: Multimodal Large Language Models (MLLMs) have experienced rapid progress in
  visual recognition tasks in recent years. Given their potential integration
  into many critical applications, it is important to understand the limitations
  of their visual perception. In this work, we study whether MLLMs can perceive
  small visual details as effectively as large ones when answering questions
  about images. We observe that their performance is very sensitive to the size
  of the visual subject of the question, and further show that this effect is in
  fact causal by conducting an intervention study. Next, we study the attention
  patterns of MLLMs when answering visual questions, and intriguingly find that
  they consistently know where to look, even when they provide the wrong answer.
  Based on these findings, we then propose training-free visual intervention
  methods that leverage the internal knowledge of any MLLM itself, in the form of
  attention and gradient maps, to enhance its perception of small visual details.
  We evaluate our proposed methods on two widely-used MLLMs and seven visual
  question answering benchmarks and show that they can significantly improve
  MLLMs' accuracy without requiring any training. Our results elucidate the risk
  of applying MLLMs to visual recognition tasks concerning small details and
  indicate that visual intervention using the model's internal state is a
  promising direction to mitigate this risk.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) visual recognition tasks visual perception visual subject attention patterns training-free visual intervention methods attention maps gradient maps visual question answering benchmarks visual intervention