FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion
<|reference_start|>FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion: Unified multi-model representation spaces are the foundation of multimodal understanding and generation. However, the billions of model parameters and catastrophic forgetting problems make it challenging to further enhance pre-trained unified spaces. In this work, we propose FreeBind, an idea that treats multimodal representation spaces as basic units, and freely augments pre-trained unified space by integrating knowledge from extra expert spaces via "space bonds". Specifically, we introduce two kinds of basic space bonds: 1) Space Displacement Bond and 2) Space Combination Bond. Based on these basic bonds, we design Complex Sequential & Parallel Bonds to effectively integrate multiple spaces simultaneously. Benefiting from the modularization concept, we further propose a coarse-to-fine customized inference strategy to flexibly adjust the enhanced unified space for different purposes. Experimentally, we bind ImageBind with extra image-text and audio-text expert spaces, resulting in three main variants: ImageBind++, InternVL_IB, and InternVL_IB++. These resulting spaces outperform ImageBind on 5 audio-image-text downstream tasks across 9 datasets. Moreover, via customized inference, it even surpasses the advanced audio-text and image-text expert spaces.<|reference_end|>
@article{wang2024freebind:,
    title={FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion},
    author={Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu,
  Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, Zhou Zhao},
    journal={arXiv preprint arXiv:2405.04883},
    year={2024},
    archivePrefix={arXiv},
    eprint={2405.04883},
    primaryClass={cs.CV cs.AI cs.LG}
}