Hyperbolic Learning with Synthetic Captions for Open-World Detection
<|reference_start|>Hyperbolic Learning with Synthetic Captions for Open-World Detection: Open-world detection poses significant challenges, as it requires the detection of any object using either object class labels or free-form texts. Existing related works often use large-scale manual annotated caption datasets for training, which are extremely expensive to collect. Instead, we propose to transfer knowledge from vision-language models (VLMs) to enrich the open-vocabulary descriptions automatically. Specifically, we bootstrap dense synthetic captions using pre-trained VLMs to provide rich descriptions on different regions in images, and incorporate these captions to train a novel detector that generalizes to novel concepts. To mitigate the noise caused by hallucination in synthetic captions, we also propose a novel hyperbolic vision-language learning approach to impose a hierarchy between visual and caption embeddings. We call our detector ``HyperLearner''. We conduct extensive experiments on a wide variety of open-world detection benchmarks (COCO, LVIS, Object Detection in the Wild, RefCOCO) and our results show that our model consistently outperforms existing state-of-the-art methods, such as GLIP, GLIPv2 and Grounding DINO, when using the same backbone.<|reference_end|>
@article{kong2024hyperbolic,
    title={Hyperbolic Learning with Synthetic Captions for Open-World Detection},
    author={Fanjie Kong, Yanbei Chen, Jiarui Cai, Davide Modolo},
    journal={arXiv preprint arXiv:2404.05016},
    year={2024},
    archivePrefix={arXiv},
    eprint={2404.05016},
    primaryClass={cs.CV}
}