title: Improve Vision Language Model Chain-of-thought Reasoning
summary: Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial
  for improving interpretability and trustworthiness. However, current training
  recipes lack robust CoT reasoning data, relying on datasets dominated by short
  annotations with minimal rationales. In this work, we show that training VLM on
  short answers does not generalize well to reasoning tasks that require more
  detailed responses. To address this, we propose a two-fold approach. First, we
  distill rationales from GPT-4o model to enrich the training data and fine-tune
  VLMs, boosting their CoT performance. Second, we apply reinforcement learning
  to further calibrate reasoning quality. Specifically, we construct positive
  (correct) and negative (incorrect) pairs of model-generated reasoning chains,
  by comparing their predictions with annotated short answers. Using this
  pairwise data, we apply the Direct Preference Optimization algorithm to refine
  the model's reasoning abilities. Our experiments demonstrate significant
  improvements in CoT reasoning on benchmark datasets and better generalization
  to direct answer prediction as well. This work emphasizes the importance of
  incorporating detailed rationales in training and leveraging reinforcement
  learning to strengthen the reasoning capabilities of VLMs.
paper_authors._id: 67189521cba378ccb674f2e9
paper_authors.name: Ruohong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:46.100Z
paper_authors._id: 63eeb0bbbe5d5900b85130ca
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63eeb0bbbe5d5900b85130ca/ofmp91Zp7xAppQACCKz4L.jpeg
paper_authors.fullname: Ruohong Zhang
paper_authors.type: user
paper_authors.user: ruohongz
paper_authors._id: 67189521cba378ccb674f2ea
paper_authors.name: Bowen Zhang
paper_authors._id: 67189521cba378ccb674f2eb
paper_authors.name: Yanghao Li
paper_authors._id: 67189521cba378ccb674f2ec
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-24T07:15:43.267Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 67189521cba378ccb674f2ed
paper_authors.name: Zhiqing Sun
paper_authors._id: 67189521cba378ccb674f2ee
paper_authors.name: Zhe Gan
paper_authors._id: 67189521cba378ccb674f2ef
paper_authors.name: Yinfei Yang
paper_authors._id: 67189521cba378ccb674f2f0
paper_authors.name: Ruoming Pang
paper_authors._id: 67189521cba378ccb674f2f1
paper_authors.name: Yiming Yang
paper_title: Improve Vision Language Model Chain-of-thought Reasoning
paper_summary: Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial
  for improving interpretability and trustworthiness. However, current training
  recipes lack robust CoT reasoning data, relying on datasets dominated by short
  annotations with minimal rationales. In this work, we show that training VLM on
  short answers does not generalize well to reasoning tasks that require more
  detailed responses. To address this, we propose a two-fold approach. First, we
  distill rationales from GPT-4o model to enrich the training data and fine-tune
  VLMs, boosting their CoT performance. Second, we apply reinforcement learning
  to further calibrate reasoning quality. Specifically, we construct positive
  (correct) and negative (incorrect) pairs of model-generated reasoning chains,
  by comparing their predictions with annotated short answers. Using this
  pairwise data, we apply the Direct Preference Optimization algorithm to refine
  the model's reasoning abilities. Our experiments demonstrate significant
  improvements in CoT reasoning on benchmark datasets and better generalization
  to direct answer prediction as well. This work emphasizes the importance of
  incorporating detailed rationales in training and leveraging reinforcement
  learning to strengthen the reasoning capabilities of VLMs.
paper_ai_summary: Enhancing vision language models with detailed rationales through distillation and reinforcement learning improves their chain-of-thought reasoning and generalization.
paper_ai_keywords: VLMs chain-of-thought reasoning distillation reinforcement learning Direct Preference Optimization annotation fine-tuning