Question Aware Vision Transformer for Multimodal Reasoning
Vision-Language (VL) models have gained significant research focus, enabling
remarkable advances in multimodal reasoning. These architectures typically
comprise a vision encoder, a Large Language Model (LLM), and a projection
module that aligns visual features with the LLM's representation space. Despite
their success, a critical limitation persists: the vision encoding process
remains decoupled from user queries, often in the form of image-related
questions. Consequently, the resulting visual features may not be optimally
attuned to the query-specific elements of the image. To address this, we
introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
reasoning, which embeds question awareness directly within the vision encoder.
This integration results in dynamic visual features focusing on relevant image
aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
efficiently into any VL architecture. Extensive experiments demonstrate the
effectiveness of applying our method to various multimodal architectures,
leading to consistent improvement across diverse tasks and showcasing its
potential for enhancing visual and scene-text understanding.
65c595f299c74418883df8c4
Roy Ganz
admin_assigned
2024-02-09T10:29:22.892Z
62f0cb39671fc964b5063aba
/avatars/02404cd78c395331b42500dd6ced35eb.svg
Roy Ganz
user
proy
65c595f299c74418883df8c5
Yair Kittenplon
65c595f299c74418883df8c6
Aviad Aberdam
65c595f299c74418883df8c7
Elad Ben Avraham
admin_assigned
2024-02-09T10:29:35.361Z
640dea72c364a086c632cf8a
/avatars/81fa19e48d18eebb51f528be6f83e64e.svg
Elad Ben Avraham
user
eladba
65c595f299c74418883df8c8
Oren Nuriel
65c595f299c74418883df8c9
Shai Mazor
65c595f299c74418883df8ca
Ron Litman
Question Aware Vision Transformer for Multimodal Reasoning
Vision-Language (VL) models have gained significant research focus, enabling
remarkable advances in multimodal reasoning. These architectures typically
comprise a vision encoder, a Large Language Model (LLM), and a projection
module that aligns visual features with the LLM's representation space. Despite
their success, a critical limitation persists: the vision encoding process
remains decoupled from user queries, often in the form of image-related
questions. Consequently, the resulting visual features may not be optimally
attuned to the query-specific elements of the image. To address this, we
introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal
reasoning, which embeds question awareness directly within the vision encoder.
This integration results in dynamic visual features focusing on relevant image
aspects to the posed question. QA-ViT is model-agnostic and can be incorporated
efficiently into any VL architecture. Extensive experiments demonstrate the
effectiveness of applying our method to various multimodal architectures,
leading to consistent improvement across diverse tasks and showcasing its
potential for enhancing visual and scene-text understanding.
QA-ViT enhances multimodal reasoning by integrating question awareness directly into the vision encoder, resulting in improved visual and scene-text understanding.
vision-language models multimodal reasoning vision encoder Large Language Model (LLM) projection module question awareness vision transformer multimodal architectures visual features scene-text understanding