title: Question Aware Vision Transformer for Multimodal Reasoning
summary: Vision-Language (VL) models have gained significant research focus, enabling
  remarkable advances in multimodal reasoning. These architectures typically
  comprise a vision encoder, a Large Language Model (LLM), and a projection
  module that aligns visual features with the LLM's representation space. Despite
  their success, a critical limitation persists: the vision encoding process
  remains decoupled from user queries, often in the form of image-related
  questions. Consequently, the resulting visual features may not be optimally
  attuned to the query-specific elements of the image. To address this, we
  introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
  reasoning, which embeds question awareness directly within the vision encoder.
  This integration results in dynamic visual features focusing on relevant image
  aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
  efficiently into any VL architecture. Extensive experiments demonstrate the
  effectiveness of applying our method to various multimodal architectures,
  leading to consistent improvement across diverse tasks and showcasing its
  potential for enhancing visual and scene-text understanding.
paper_authors._id: 65c595f299c74418883df8c4
paper_authors.name: Roy Ganz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:29:22.892Z
paper_authors._id: 62f0cb39671fc964b5063aba
paper_authors.avatarUrl: /avatars/02404cd78c395331b42500dd6ced35eb.svg
paper_authors.fullname: Roy Ganz
paper_authors.type: user
paper_authors.user: proy
paper_authors._id: 65c595f299c74418883df8c5
paper_authors.name: Yair Kittenplon
paper_authors._id: 65c595f299c74418883df8c6
paper_authors.name: Aviad Aberdam
paper_authors._id: 65c595f299c74418883df8c7
paper_authors.name: Elad Ben Avraham
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:29:35.361Z
paper_authors._id: 640dea72c364a086c632cf8a
paper_authors.avatarUrl: /avatars/81fa19e48d18eebb51f528be6f83e64e.svg
paper_authors.fullname: Elad Ben Avraham
paper_authors.type: user
paper_authors.user: eladba
paper_authors._id: 65c595f299c74418883df8c8
paper_authors.name: Oren Nuriel
paper_authors._id: 65c595f299c74418883df8c9
paper_authors.name: Shai Mazor
paper_authors._id: 65c595f299c74418883df8ca
paper_authors.name: Ron Litman
paper_title: Question Aware Vision Transformer for Multimodal Reasoning
paper_summary: Vision-Language (VL) models have gained significant research focus, enabling
  remarkable advances in multimodal reasoning. These architectures typically
  comprise a vision encoder, a Large Language Model (LLM), and a projection
  module that aligns visual features with the LLM's representation space. Despite
  their success, a critical limitation persists: the vision encoding process
  remains decoupled from user queries, often in the form of image-related
  questions. Consequently, the resulting visual features may not be optimally
  attuned to the query-specific elements of the image. To address this, we
  introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
  reasoning, which embeds question awareness directly within the vision encoder.
  This integration results in dynamic visual features focusing on relevant image
  aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
  efficiently into any VL architecture. Extensive experiments demonstrate the
  effectiveness of applying our method to various multimodal architectures,
  leading to consistent improvement across diverse tasks and showcasing its
  potential for enhancing visual and scene-text understanding.
paper_ai_summary: QA-ViT enhances multimodal reasoning by integrating question awareness directly into the vision encoder, resulting in improved visual and scene-text understanding.
paper_ai_keywords: vision-language models multimodal reasoning vision encoder Large Language Model (LLM) projection module question awareness vision transformer multimodal architectures visual features scene-text understanding