Grounding Language Models for Visual Entity Recognition
<|reference_start|>Grounding Language Models for Visual Entity Recognition: We introduce AutoVER, an Autoregressive model for Visual Entity Recognition. Our model extends an autoregressive Multi-modal Large Language Model by employing retrieval augmented constrained generation. It mitigates low performance on out-of-domain entities while excelling in queries that require visually-situated reasoning. Our method learns to distinguish similar entities within a vast label space by contrastively training on hard negative pairs in parallel with a sequence-to-sequence objective without an external retriever. During inference, a list of retrieved candidate answers explicitly guides language generation by removing invalid decoding paths. The proposed method achieves significant improvements across different dataset splits in the recently proposed Oven-Wiki benchmark. Accuracy on the Entity seen split rises from 32.7% to 61.5%. It also demonstrates superior performance on the unseen and query splits by a substantial double-digit margin.<|reference_end|>
@article{xiao2024grounding,
    title={Grounding Language Models for Visual Entity Recognition},
    author={Zilin Xiao, Ming Gong, Paola Cascante-Bonilla, Xingyao Zhang, Jie Wu,
  Vicente Ordonez},
    journal={arXiv preprint arXiv:2402.18695},
    year={2024},
    archivePrefix={arXiv},
    eprint={2402.18695},
    primaryClass={cs.CV cs.CL}
}