Improve Vision Language Model Chain-of-thought Reasoning
Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial
for improving interpretability and trustworthiness. However, current training
recipes lack robust CoT reasoning data, relying on datasets dominated by short
annotations with minimal rationales. In this work, we show that training VLM on
short answers does not generalize well to reasoning tasks that require more
detailed responses. To address this, we propose a two-fold approach. First, we
distill rationales from GPT-4o model to enrich the training data and fine-tune
VLMs, boosting their CoT performance. Second, we apply reinforcement learning
to further calibrate reasoning quality. Specifically, we construct positive
(correct) and negative (incorrect) pairs of model-generated reasoning chains,
by comparing their predictions with annotated short answers. Using this
pairwise data, we apply the Direct Preference Optimization algorithm to refine
the model's reasoning abilities. Our experiments demonstrate significant
improvements in CoT reasoning on benchmark datasets and better generalization
to direct answer prediction as well. This work emphasizes the importance of
incorporating detailed rationales in training and leveraging reinforcement
learning to strengthen the reasoning capabilities of VLMs.
67189521cba378ccb674f2e9
Ruohong Zhang
claimed_verified
2024-10-24T07:15:46.100Z
63eeb0bbbe5d5900b85130ca
https://cdn-avatars.huggingface.co/v1/production/uploads/63eeb0bbbe5d5900b85130ca/ofmp91Zp7xAppQACCKz4L.jpeg
Ruohong Zhang
user
ruohongz
67189521cba378ccb674f2ea
Bowen Zhang
67189521cba378ccb674f2eb
Yanghao Li
67189521cba378ccb674f2ec
Haotian Zhang
claimed_verified
2024-10-24T07:15:43.267Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
67189521cba378ccb674f2ed
Zhiqing Sun
67189521cba378ccb674f2ee
Zhe Gan
67189521cba378ccb674f2ef
Yinfei Yang
67189521cba378ccb674f2f0
Ruoming Pang
67189521cba378ccb674f2f1
Yiming Yang
Improve Vision Language Model Chain-of-thought Reasoning
Chain-of-thought (CoT) reasoning in vision language models (VLMs) is crucial
for improving interpretability and trustworthiness. However, current training
recipes lack robust CoT reasoning data, relying on datasets dominated by short
annotations with minimal rationales. In this work, we show that training VLM on
short answers does not generalize well to reasoning tasks that require more
detailed responses. To address this, we propose a two-fold approach. First, we
distill rationales from GPT-4o model to enrich the training data and fine-tune
VLMs, boosting their CoT performance. Second, we apply reinforcement learning
to further calibrate reasoning quality. Specifically, we construct positive
(correct) and negative (incorrect) pairs of model-generated reasoning chains,
by comparing their predictions with annotated short answers. Using this
pairwise data, we apply the Direct Preference Optimization algorithm to refine
the model's reasoning abilities. Our experiments demonstrate significant
improvements in CoT reasoning on benchmark datasets and better generalization
to direct answer prediction as well. This work emphasizes the importance of
incorporating detailed rationales in training and leveraging reinforcement
learning to strengthen the reasoning capabilities of VLMs.
Enhancing vision language models with detailed rationales through distillation and reinforcement learning improves their chain-of-thought reasoning and generalization.
VLMs chain-of-thought reasoning distillation reinforcement learning Direct Preference Optimization annotation fine-tuning