RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection
<|reference_start|>RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection: Open-vocabulary object detection (OVD) requires solid modeling of the region-semantic relationship, which could be learned from massive region-text pairs. However, such data is limited in practice due to significant annotation costs. In this work, we propose RTGen to generate scalable open-vocabulary region-text pairs and demonstrate its capability to boost the performance of open-vocabulary object detection. RTGen includes both text-to-region and region-to-text generation processes on scalable image-caption data. The text-to-region generation is powered by image inpainting, directed by our proposed scene-aware inpainting guider for overall layout harmony. For region-to-text generation, we perform multiple region-level image captioning with various prompts and select the best matching text according to CLIP similarity. To facilitate detection training on region-text pairs, we also introduce a localization-aware region-text contrastive loss that learns object proposals tailored with different localization qualities. Extensive experiments demonstrate that our RTGen can serve as a scalable, semantically rich, and effective source for open-vocabulary object detection and continue to improve the model performance when more data is utilized, delivering superior performance compared to the existing state-of-the-art methods.<|reference_end|>
@article{chen2024rtgen:,
    title={RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detection},
    author={Fangyi Chen, Han Zhang, Zhantao Yang, Hao Chen, Kai Hu and Marios
  Savvides},
    journal={arXiv preprint arXiv:2405.19854},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.19854},
    primaryClass={cs.CV}
}