Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks
<|reference_start|>Self-Correction is More than Refinement: A Learning Framework for Visual and Language Reasoning Tasks: While Vision-Language Models (VLMs) have shown remarkable abilities in visual and language reasoning tasks, they invariably generate flawed responses. Self-correction that instructs models to refine their outputs presents a promising solution to this issue. Previous studies have mainly concentrated on Large Language Models (LLMs), while the self-correction abilities of VLMs, particularly concerning both visual and linguistic information, remain largely unexamined. This study investigates the self-correction capabilities of VLMs during both inference and fine-tuning stages. We introduce a Self-Correction Learning (SCL) approach that enables VLMs to learn from their self-generated self-correction data through Direct Preference Optimization (DPO) without relying on external feedback, facilitating self-improvement. Specifically, we collect preferred and disfavored samples based on the correctness of initial and refined responses, which are obtained by two-turn self-correction with VLMs during the inference stage. Experimental results demonstrate that although VLMs struggle to self-correct effectively during iterative inference without additional fine-tuning and external feedback, they can enhance their performance and avoid previous mistakes through preference fine-tuning when their self-generated self-correction data are categorized into preferred and disfavored samples. This study emphasizes that self-correction is not merely a refinement process; rather, it should enhance the reasoning abilities of models through additional training, enabling them to generate high-quality responses directly without further refinement.<|reference_end|>
@article{he2024self-correction,
    title={Self-Correction is More than Refinement: A Learning Framework for Visual
  and Language Reasoning Tasks},
    author={Jiayi He, Hehai Lin, Qingyun Wang, Yi Fung, Heng Ji},
    journal={arXiv preprint arXiv:2410.04055},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.04055},
    primaryClass={cs.CL}
}