title: Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
summary: Vision-language models~(VLMs) have shown remarkable advancements in
  multimodal reasoning tasks. However, they still often generate inaccurate or
  irrelevant responses due to issues like hallucinated image understandings or
  unrefined reasoning paths. To address these challenges, we introduce Critic-V,
  a novel framework inspired by the Actor-Critic paradigm to boost the reasoning
  capability of VLMs. This framework decouples the reasoning process and critic
  process by integrating two independent components: the Reasoner, which
  generates reasoning paths based on visual and textual inputs, and the Critic,
  which provides constructive critique to refine these paths. In this approach,
  the Reasoner generates reasoning responses according to text prompts, which can
  evolve iteratively as a policy based on feedback from the Critic. This
  interaction process was theoretically driven by a reinforcement learning
  framework where the Critic offers natural language critiques instead of scalar
  rewards, enabling more nuanced feedback to boost the Reasoner's capability on
  complex reasoning tasks. The Critic model is trained using Direct Preference
  Optimization (DPO), leveraging a preference dataset of critiques ranked by
  Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results
  show that the Critic-V framework significantly outperforms existing methods,
  including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning
  accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner
  and constructive feedback from the preference-optimized Critic enables a more
  reliable and context-sensitive multimodal reasoning process. Our approach
  provides a promising solution to enhance the reliability of VLMs, improving
  their performance in real-world reasoning-heavy multimodal applications such as
  autonomous driving and embodied intelligence.
paper_authors._id: 67481f451cf82b1a47ba94f2
paper_authors.name: Di Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:17.745Z
paper_authors._id: 64bce15bafd1e46c5504ad38
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64bce15bafd1e46c5504ad38/bQFX1iFbXEBXcQvUNL811.png
paper_authors.fullname: Di Zhang
paper_authors.type: user
paper_authors.user: di-zhang-fdu
paper_authors._id: 67481f451cf82b1a47ba94f3
paper_authors.name: Jingdi Lei
paper_authors._id: 67481f451cf82b1a47ba94f4
paper_authors.name: Junxian Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:13.767Z
paper_authors._id: 656ae4088fb1ddf0d5ec9ac5
paper_authors.avatarUrl: /avatars/e38468d2c0274f3c0f5732f30a2e3436.svg
paper_authors.fullname: Junxian Li
paper_authors.type: user
paper_authors.user: Duke-de-Artois
paper_authors._id: 67481f451cf82b1a47ba94f5
paper_authors.name: Xunzhi Wang
paper_authors._id: 67481f451cf82b1a47ba94f6
paper_authors.name: Yujie Liu
paper_authors._id: 67481f451cf82b1a47ba94f7
paper_authors.name: Zonglin Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T11:38:00.768Z
paper_authors._id: 646a11791556443f24b582e9
paper_authors.avatarUrl: /avatars/b54d416ddca2f124492ba51bc4b95fd2.svg
paper_authors.fullname: Zonglin Yang
paper_authors.type: user
paper_authors.user: ZonglinY
paper_authors._id: 67481f451cf82b1a47ba94f8
paper_authors.name: Jiatong Li
paper_authors._id: 67481f451cf82b1a47ba94f9
paper_authors.name: Weida Wang
paper_authors._id: 67481f451cf82b1a47ba94fa
paper_authors.name: Suorong Yang
paper_authors._id: 67481f451cf82b1a47ba94fb
paper_authors.name: Jianbo Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-28T09:19:16.199Z
paper_authors._id: 6407d3d3a7bc7c3865addc4b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6407d3d3a7bc7c3865addc4b/NEMP6xShRq-5H_rQ5ganY.jpeg
paper_authors.fullname: Jianbo Wu
paper_authors.type: user
paper_authors.user: jwu323
paper_authors._id: 67481f451cf82b1a47ba94fc
paper_authors.name: Peng Ye
paper_authors._id: 67481f451cf82b1a47ba94fd
paper_authors.name: Wanli Ouyang
paper_authors._id: 67481f451cf82b1a47ba94fe
paper_authors.name: Dongzhan Zhou
paper_title: Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning
paper_summary: Vision-language models~(VLMs) have shown remarkable advancements in
  multimodal reasoning tasks. However, they still often generate inaccurate or
  irrelevant responses due to issues like hallucinated image understandings or
  unrefined reasoning paths. To address these challenges, we introduce Critic-V,
  a novel framework inspired by the Actor-Critic paradigm to boost the reasoning
  capability of VLMs. This framework decouples the reasoning process and critic
  process by integrating two independent components: the Reasoner, which
  generates reasoning paths based on visual and textual inputs, and the Critic,
  which provides constructive critique to refine these paths. In this approach,
  the Reasoner generates reasoning responses according to text prompts, which can
  evolve iteratively as a policy based on feedback from the Critic. This
  interaction process was theoretically driven by a reinforcement learning
  framework where the Critic offers natural language critiques instead of scalar
  rewards, enabling more nuanced feedback to boost the Reasoner's capability on
  complex reasoning tasks. The Critic model is trained using Direct Preference
  Optimization (DPO), leveraging a preference dataset of critiques ranked by
  Rule-based Reward(RBR) to enhance its critic capabilities. Evaluation results
  show that the Critic-V framework significantly outperforms existing methods,
  including GPT-4V, on 5 out of 8 benchmarks, especially regarding reasoning
  accuracy and efficiency. Combining a dynamic text-based policy for the Reasoner
  and constructive feedback from the preference-optimized Critic enables a more
  reliable and context-sensitive multimodal reasoning process. Our approach
  provides a promising solution to enhance the reliability of VLMs, improving
  their performance in real-world reasoning-heavy multimodal applications such as
  autonomous driving and embodied intelligence.
paper_ai_keywords: Vision-language models (VLMs) multimodal reasoning tasks hallucinated image understandings unrefined reasoning paths Actor-Critic paradigm Reasoner Critic reasoning paths text prompts policy feedback reinforcement learning Direct Preference Optimization (DPO) preference dataset Rule-based Reward (RBR) reasoning accuracy efficiency dynamic text-based policy preference-optimized Critic reliable and context-sensitive multimodal reasoning process real-world reasoning-heavy multimodal applications autonomous driving embodied intelligence