VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models
<|reference_start|>VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models: While large multi-modal models (LMMs) have exhibited impressive capabilities across diverse tasks, their effectiveness in handling complex tasks has been limited by the prevailing single-step reasoning paradigm. To this end, this paper proposes VoCoT, a multi-step Visually grounded object-centric Chain-of-Thought reasoning framework tailored for inference with LMMs. VoCoT is characterized by two key features: (1) object-centric reasoning paths that revolve around cross-modal shared object-level information, and (2) visually grounded representation of object concepts in a multi-modal interleaved and aligned manner, which effectively bridges the modality gap within LMMs during long-term generation. Additionally, we construct an instruction dataset to facilitate LMMs in adapting to reasoning with VoCoT. By introducing VoCoT into the prevalent open-source LMM architecture, we introduce VolCano. With only 7B parameters and limited input resolution, VolCano demonstrates excellent performance across various scenarios, surpassing SOTA models, including GPT-4V, in tasks requiring complex reasoning. Our code, data and model will be available at https://github.com/RupertLuo/VoCoT.<|reference_end|>
@article{li2024vocot:,
    title={VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large
  Multi-Modal Models},
    author={Zejun Li, Ruipu Luo, Jiwen Zhang, Minghui Qiu, Zhongyu Wei},
    journal={arXiv preprint arXiv:2405.16919},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.16919},
    primaryClass={cs.CV cs.AI cs.CL}
}