ParGo: Bridging Vision-Language with Partial and Global Views
<|reference_start|>ParGo: Bridging Vision-Language with Partial and Global Views: This work presents ParGo, a novel Partial-Global projector designed to connect the vision and language modalities for Multimodal Large Language Models (MLLMs). Unlike previous works that rely on global attention-based projectors, our ParGo bridges the representation gap between the separately pre-trained vision encoders and the LLMs by integrating global and partial views, which alleviates the overemphasis on prominent regions. To facilitate the effective training of ParGo, we collect a large-scale detail-captioned image-text dataset named ParGoCap-1M-PT, consisting of 1 million images paired with high-quality captions. Extensive experiments on several MLLM benchmarks demonstrate the effectiveness of our ParGo, highlighting its superiority in aligning vision and language modalities. Compared to conventional Q-Former projector, our ParGo achieves an improvement of 259.96 in MME benchmark. Furthermore, our experiments reveal that ParGo significantly outperforms other projectors, particularly in tasks that emphasize detail perception ability.<|reference_end|>
@article{wang2024pargo:,
    title={ParGo: Bridging Vision-Language with Partial and Global Views},
    author={An-Lan Wang, Bin Shan, Wei Shi, Kun-Yu Lin, Xiang Fei, Guozhi Tang,
  Lei Liao, Jingqun Tang, Can Huang, Wei-Shi Zheng},
    journal={arXiv preprint arXiv:2408.12928},
    year={2024},
    archivePrefix={arXiv},
    eprint={2408.12928},
    primaryClass={cs.CV}
}