Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition
<|reference_start|>Multi-Grained Query-Guided Set Prediction Network for Grounded Multimodal Named Entity Recognition: Grounded Multimodal Named Entity Recognition (GMNER) is an emerging information extraction (IE) task, aiming to simultaneously extract entity spans, types, and corresponding visual regions of entities from given sentence-image pairs data. Recent unified methods employing machine reading comprehension or sequence generation-based frameworks show limitations in this difficult task. The former, utilizing human-designed queries, struggles to differentiate ambiguous entities, such as Jordan (Person) and off-White x Jordan (Shoes). The latter, following the one-by-one decoding order, suffers from exposure bias issues. We maintain that these works misunderstand the relationships of multimodal entities. To tackle these, we propose a novel unified framework named Multi-grained Query-guided Set Prediction Network (MQSPN) to learn appropriate relationships at intra-entity and inter-entity levels. Specifically, MQSPN consists of a Multi-grained Query Set (MQS) and a Multimodal Set Prediction Network (MSP). MQS explicitly aligns entity regions with entity spans by employing a set of learnable queries to strengthen intra-entity connections. Based on distinct intra-entity modeling, MSP reformulates GMNER as a set prediction, guiding models to establish appropriate inter-entity relationships from a global matching perspective. Additionally, we incorporate a query-guided Fusion Net (QFNet) to work as a glue network between MQS and MSP. Extensive experiments demonstrate that our approach achieves state-of-the-art performances in widely used benchmarks.<|reference_end|>
@article{tang2024multi-grained,
    title={Multi-Grained Query-Guided Set Prediction Network for Grounded
  Multimodal Named Entity Recognition},
    author={Jielong Tang, Zhenxing Wang, Ziyang Gong, Jianxing Yu, Xiangwei Zhu
  and Jian Yin},
    journal={arXiv preprint arXiv:2407.21033},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.21033},
    primaryClass={cs.IR cs.AI cs.CL cs.CV}
}