Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations
<|reference_start|>Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations: Multimodal learning plays a crucial role in enabling machine learning models to fuse and utilize diverse data sources, such as text, images, and audio, to support a variety of downstream tasks. A unified representation across various modalities is particularly important for improving efficiency and performance. Recent binding methods, such as ImageBind (Girdhar et al., 2023), typically use a fixed anchor modality to align multimodal data in the anchor modal embedding space. In this paper, we mathematically analyze the fixed anchor binding methods and uncover notable limitations: (1) over-reliance on the choice of the anchor modality, (2) failure to capture intra-modal information, and (3) failure to account for inter-modal correlation among non-anchored modalities. To address these limitations, we propose CentroBind, a simple yet powerful approach that eliminates the need for a fixed anchor; instead, it employs dynamically adjustable centroid-based anchors generated from all available modalities, resulting in a balanced and rich representation space. We theoretically demonstrate that our method captures three crucial properties of multimodal learning: intra-modal learning, inter-modal learning, and multimodal alignment, while also constructing a robust unified representation across all modalities. Our experiments on both synthetic and real-world datasets demonstrate the superiority of the proposed method, showing that dynamic anchor methods outperform all fixed anchor binding methods as the former captures more nuanced multimodal interactions.<|reference_end|>
@article{jeong2024anchors,
    title={Anchors Aweigh! Sail for Optimal Unified Multi-Modal Representations},
    author={Minoh Jeong, Min Namgung, Zae Myung Kim, Dongyeop Kang, Yao-Yi Chiang,
  Alfred Hero},
    journal={arXiv preprint arXiv:2410.02086},
    year={2024},
    archivePrefix={arXiv},
    eprint={2410.02086},
    primaryClass={cs.LG cs.CV stat.ML}
}