title: Visual Question Decomposition on Multimodal Large Language Models
summary: Question decomposition has emerged as an effective strategy for prompting
  Large Language Models (LLMs) to answer complex questions. However, while
  existing methods primarily focus on unimodal language models, the question
  decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
  be explored. To this end, this paper explores visual question decomposition on
  MLLMs. Specifically, we introduce a systematic evaluation framework including a
  dataset and several evaluation criteria to assess the quality of the decomposed
  sub-questions, revealing that existing MLLMs struggle to produce high-quality
  sub-questions. To address this limitation, we propose a specific finetuning
  dataset, DecoVQA+, for enhancing the model's question decomposition capability.
  Aiming at enabling models to perform appropriate selective decomposition, we
  propose an efficient finetuning pipeline. The finetuning pipeline consists of
  our proposed dataset and a training objective for selective decomposition.
  Finetuned MLLMs demonstrate significant improvements in the quality of
  sub-questions and the policy of selective question decomposition. Additionally,
  the models also achieve higher accuracy with selective decomposition on VQA
  benchmark datasets.
paper_authors._id: 66fbeeb0f553900c02224daa
paper_authors.name: Haowei Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:20.732Z
paper_authors._id: 637169557a5e5d8efdc3e58e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1668515232215-637169557a5e5d8efdc3e58e.jpeg
paper_authors.fullname: Haowei Zhang
paper_authors.type: user
paper_authors.user: freesky
paper_authors._id: 66fbeeb0f553900c02224dab
paper_authors.name: Jianzhe Liu
paper_authors._id: 66fbeeb0f553900c02224dac
paper_authors.name: Zhen Han
paper_authors._id: 66fbeeb0f553900c02224dad
paper_authors.name: Shuo Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-02T07:41:22.914Z
paper_authors._id: 648cbea3dee03837c823cbf2
paper_authors.avatarUrl: /avatars/3f8c36436a5cbff2948df099ae604418.svg
paper_authors.fullname: Shuo Chen
paper_authors.type: user
paper_authors.user: ShuoChen99
paper_authors._id: 66fbeeb0f553900c02224dae
paper_authors.name: Bailan He
paper_authors._id: 66fbeeb0f553900c02224daf
paper_authors.name: Volker Tresp
paper_authors._id: 66fbeeb0f553900c02224db0
paper_authors.name: Zhiqiang Xu
paper_authors._id: 66fbeeb0f553900c02224db1
paper_authors.name: Jindong Gu
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-11T12:13:42.512Z
paper_authors._id: 67d0159c0fc5d2bf9502c59a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/sOZLe-APA6UHPKci7P-C9.png
paper_authors.fullname: Gu
paper_authors.type: user
paper_authors.user: Jindong01
paper_title: Visual Question Decomposition on Multimodal Large Language Models
paper_summary: Question decomposition has emerged as an effective strategy for prompting
  Large Language Models (LLMs) to answer complex questions. However, while
  existing methods primarily focus on unimodal language models, the question
  decomposition capability of Multimodal Large Language Models (MLLMs) has yet to
  be explored. To this end, this paper explores visual question decomposition on
  MLLMs. Specifically, we introduce a systematic evaluation framework including a
  dataset and several evaluation criteria to assess the quality of the decomposed
  sub-questions, revealing that existing MLLMs struggle to produce high-quality
  sub-questions. To address this limitation, we propose a specific finetuning
  dataset, DecoVQA+, for enhancing the model's question decomposition capability.
  Aiming at enabling models to perform appropriate selective decomposition, we
  propose an efficient finetuning pipeline. The finetuning pipeline consists of
  our proposed dataset and a training objective for selective decomposition.
  Finetuned MLLMs demonstrate significant improvements in the quality of
  sub-questions and the policy of selective question decomposition. Additionally,
  the models also achieve higher accuracy with selective decomposition on VQA
  benchmark datasets.
paper_ai_keywords: Federated Learning (FL) Large Language Models (LLMs) Multimodal Large Language Models (MLLMs) visual question decomposition evaluation framework dataset evaluation criteria finetuning dataset DecoVQA+ selective decomposition finetuning pipeline training objective selective question decomposition VQA benchmark datasets