Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media
<|reference_start|>Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media: Semantic location prediction aims to derive meaningful location insights from multimodal social media posts, offering a more contextual understanding of daily activities than using GPS coordinates. This task faces significant challenges due to the noise and modality heterogeneity in "text-image" posts. Existing methods are generally constrained by inadequate feature representations and modal interaction, struggling to effectively reduce noise and modality heterogeneity. To address these challenges, we propose a Similarity-Guided Multimodal Fusion Transformer (SG-MFT) for predicting the semantic locations of users from their multimodal posts. First, we incorporate high-quality text and image representations by utilizing a pre-trained large vision-language model. Then, we devise a Similarity-Guided Interaction Module (SIM) to alleviate modality heterogeneity and noise interference by incorporating both coarse-grained and fine-grained similarity guidance for improving modality interactions. Specifically, we propose a novel similarity-aware feature interpolation attention mechanism at the coarse-grained level, leveraging modality-wise similarity to mitigate heterogeneity and reduce noise within each modality. At the fine-grained level, we utilize a similarity-aware feed-forward block and element-wise similarity to further address the issue of modality heterogeneity. Finally, building upon pre-processed features with minimal noise and modal interference, we devise a Similarity-aware Fusion Module (SFM) to fuse two modalities with a cross-attention mechanism. Comprehensive experimental results clearly demonstrate the superior performance of our proposed method.<|reference_end|>
@article{zhang2024similarity,
    title={Similarity Guided Multimodal Fusion Transformer for Semantic Location
  Prediction in Social Media},
    author={Zhizhen Zhang, Ning Wang, Haojie Li, Zhihui Wang},
    journal={arXiv preprint arXiv:2405.05760},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.05760},
    primaryClass={cs.CV cs.CL}
}