Visual Question Decomposition on Multimodal Large Language Models
Question decomposition has emerged as an effective strategy for prompting
Large Language Models (LLMs) to answer complex questions. However, while
existing methods primarily focus on unimodal language models, the question
decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
be explored. To this end, this paper explores visual question decomposition on
MLLMs. Specifically, we introduce a systematic evaluation framework including a
dataset and several evaluation criteria to assess the quality of the decomposed
sub-questions, revealing that existing MLLMs struggle to produce high-quality
sub-questions. To address this limitation, we propose a specific finetuning
dataset, DecoVQA+, for enhancing the model's question decomposition capability.
Aiming at enabling models to perform appropriate selective decomposition, we
propose an efficient finetuning pipeline. The finetuning pipeline consists of
our proposed dataset and a training objective for selective decomposition.
Finetuned MLLMs demonstrate significant improvements in the quality of
sub-questions and the policy of selective question decomposition. Additionally,
the models also achieve higher accuracy with selective decomposition on VQA
benchmark datasets.
66fbeeb0f553900c02224daa
Haowei Zhang
claimed_verified
2024-10-02T07:41:20.732Z
637169557a5e5d8efdc3e58e
https://cdn-avatars.huggingface.co/v1/production/uploads/1668515232215-637169557a5e5d8efdc3e58e.jpeg
Haowei Zhang
user
freesky
66fbeeb0f553900c02224dab
Jianzhe Liu
66fbeeb0f553900c02224dac
Zhen Han
66fbeeb0f553900c02224dad
Shuo Chen
claimed_verified
2024-10-02T07:41:22.914Z
648cbea3dee03837c823cbf2
/avatars/3f8c36436a5cbff2948df099ae604418.svg
Shuo Chen
user
ShuoChen99
66fbeeb0f553900c02224dae
Bailan He
66fbeeb0f553900c02224daf
Volker Tresp
66fbeeb0f553900c02224db0
Zhiqiang Xu
66fbeeb0f553900c02224db1
Jindong Gu
extracted_pending
2025-03-11T12:13:42.512Z
67d0159c0fc5d2bf9502c59a
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sOZLe-APA6UHPKci7P-C9.png
Gu
user
Jindong01
Visual Question Decomposition on Multimodal Large Language Models
Question decomposition has emerged as an effective strategy for prompting
Large Language Models (LLMs) to answer complex questions. However, while
existing methods primarily focus on unimodal language models, the question
decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
be explored. To this end, this paper explores visual question decomposition on
MLLMs. Specifically, we introduce a systematic evaluation framework including a
dataset and several evaluation criteria to assess the quality of the decomposed
sub-questions, revealing that existing MLLMs struggle to produce high-quality
sub-questions. To address this limitation, we propose a specific finetuning
dataset, DecoVQA+, for enhancing the model's question decomposition capability.
Aiming at enabling models to perform appropriate selective decomposition, we
propose an efficient finetuning pipeline. The finetuning pipeline consists of
our proposed dataset and a training objective for selective decomposition.
Finetuned MLLMs demonstrate significant improvements in the quality of
sub-questions and the policy of selective question decomposition. Additionally,
the models also achieve higher accuracy with selective decomposition on VQA
benchmark datasets.
An evaluation framework and finetuning pipeline for enhancing the question decomposition capability of Multimodal Large Language Models are proposed, improving sub-question quality and VQA accuracy.
Multimodal Large Language Models question decomposition evaluation framework dataset DecoVQA+ finetuning pipeline selective decomposition VQA benchmark datasets