Generalizable Entity Grounding via Assistance of Large Language Model
<|reference_start|>Generalizable Entity Grounding via Assistance of Large Language Model: In this work, we propose a novel approach to densely ground visual entities from a long caption. We leverage a large multimodal model (LMM) to extract semantic nouns, a class-agnostic segmentation model to generate entity-level segmentation, and the proposed multi-modal feature fusion module to associate each semantic noun with its corresponding segmentation mask. Additionally, we introduce a strategy of encoding entity segmentation masks into a colormap, enabling the preservation of fine-grained predictions from features of high-resolution masks. This approach allows us to extract visual features from low-resolution images using the CLIP vision encoder in the LMM, which is more computationally efficient than existing approaches that use an additional encoder for high-resolution images. Our comprehensive experiments demonstrate the superiority of our method, outperforming state-of-the-art techniques on three tasks, including panoptic narrative grounding, referring expression segmentation, and panoptic segmentation.<|reference_end|>
@article{qi2024generalizable,
    title={Generalizable Entity Grounding via Assistance of Large Language Model},
    author={Lu Qi, Yi-Wen Chen, Lehan Yang, Tiancheng Shen, Xiangtai Li, Weidong
  Guo, Yu Xu, Ming-Hsuan Yang},
    journal={arXiv preprint arXiv:2402.02555},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.02555},
    primaryClass={cs.CV cs.CL}
}