Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking
<|reference_start|>Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking: Multimodal Entity Linking (MEL) aims to link ambiguous mentions in multimodal contexts to entities in a multimodal knowledge graph. A pivotal challenge is to fully leverage multi-element correlations between mentions and entities to bridge modality gap and enable fine-grained semantic matching. Existing methods attempt several local correlative mechanisms, relying heavily on the automatically learned attention weights, which may over-concentrate on partial correlations. To mitigate this issue, we formulate the correlation assignment problem as an optimal transport (OT) problem, and propose a novel MEL framework, namely OT-MEL, with OT-guided correlation assignment. Thereby, we exploit the correlation between multimodal features to enhance multimodal fusion, and the correlation between mentions and entities to enhance fine-grained matching. To accelerate model prediction, we further leverage knowledge distillation to transfer OT assignment knowledge to attention mechanism. Experimental results show that our model significantly outperforms previous state-of-the-art baselines and confirm the effectiveness of the OT-guided correlation assignment.<|reference_end|>
@article{zhang2024optimal,
    title={Optimal Transport Guided Correlation Assignment for Multimodal Entity
  Linking},
    author={Zefeng Zhang, Jiawei Sheng, Chuang Zhang, Yunzhi Liang, Wenyuan Zhang,
  Siqi Wang, Tingwen Liu},
    journal={arXiv preprint arXiv:2406.01934},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01934},
    primaryClass={cs.CL}
}