RegionGPT: Towards Region Understanding Vision Language Model
<|reference_start|>RegionGPT: Towards Region Understanding Vision Language Model: Vision language models (VLMs) have experienced rapid advancements through the integration of large language models (LLMs) with image-text pairs, yet they struggle with detailed regional visual understanding due to limited spatial awareness of the vision encoder, and the use of coarse-grained training data that lacks detailed, region-specific captions. To address this, we introduce RegionGPT (short as RGPT), a novel framework designed for complex region-level captioning and understanding. RGPT enhances the spatial awareness of regional representation with simple yet effective modifications to existing visual encoders in VLMs. We further improve performance on tasks requiring a specific output scope by integrating task-guided instruction prompts during both training and inference phases, while maintaining the model's versatility for general-purpose tasks. Additionally, we develop an automated region caption data generation pipeline, enriching the training set with detailed region-level captions. We demonstrate that a universal RGPT model can be effectively applied and significantly enhancing performance across a range of region-level tasks, including but not limited to complex region descriptions, reasoning, object classification, and referring expressions comprehension.<|reference_end|>
@article{guo2024regiongpt:,
    title={RegionGPT: Towards Region Understanding Vision Language Model},
    author={Qiushan Guo, Shalini De Mello, Hongxu Yin, Wonmin Byeon, Ka Chun
  Cheung, Yizhou Yu, Ping Luo, Sifei Liu},
    journal={arXiv preprint arXiv:2403.02330},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.02330},
    primaryClass={cs.CV}
}