title: CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
    Multiplet Upcycling
summary: In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
  cornerstone in multimodal intelligence. However, recent studies have identified
  that the information loss in the CLIP encoding process is substantial, and CLIP
  tends to capture only coarse-grained features from the input. This deficiency
  significantly limits the ability of a single CLIP model to handle images rich
  in visual detail. In this work, we propose a simple yet effective
  model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
  efficiently fine-tunes a series of CLIP models that capture different feature
  spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
  the Feed-Forward Network (FFN). These models can then be transformed into a
  CLIP-MoE with a larger model capacity, leading to significantly enhanced
  performance with minimal computational overhead. To the best of our knowledge,
  Diversified Multiplet Upcycling is the first approach to introduce sparsely
  activated MoE into CLIP foundation models. Extensive experiments demonstrate
  the significant performance of CLIP-MoE across various zero-shot retrieval,
  zero-shot image classification tasks, and downstream Multimodal Large Language
  Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
  Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
  into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
  without requiring further adaptation in downstream frameworks. Through
  Diversified Multiplet Upcycling, we aim to provide valuable insights for future
  research on developing more efficient and effective multimodal learning
  systems.
paper_authors._id: 66ffab22dbdb4b0b9f759af9
paper_authors.name: Jihai Zhang
paper_authors._id: 66ffab22dbdb4b0b9f759afa
paper_authors.name: Xiaoye Qu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T08:46:35.159Z
paper_authors._id: 64cb54da1af278541d663708
paper_authors.avatarUrl: /avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
paper_authors.fullname: Xiaoye Qu
paper_authors.type: user
paper_authors.user: Xiaoye08
paper_authors._id: 66ffab22dbdb4b0b9f759afb
paper_authors.name: Tong Zhu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-04T09:02:32.963Z
paper_authors._id: 629454301ae2138079f7ff31
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/629454301ae2138079f7ff31/rVtbF-j06gDiYzomTeVTc.jpeg
paper_authors.fullname: Tong Zhu
paper_authors.type: user
paper_authors.user: Spico
paper_authors._id: 66ffab22dbdb4b0b9f759afc
paper_authors.name: Yu Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-09T16:26:29.239Z
paper_authors._id: 67017abfe4d49b157ac534d9
paper_authors.avatarUrl: /avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
paper_authors.fullname: Yu Cheng
paper_authors.type: user
paper_authors.user: ych133
paper_title: CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
    Multiplet Upcycling
paper_summary: In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
  cornerstone in multimodal intelligence. However, recent studies have identified
  that the information loss in the CLIP encoding process is substantial, and CLIP
  tends to capture only coarse-grained features from the input. This deficiency
  significantly limits the ability of a single CLIP model to handle images rich
  in visual detail. In this work, we propose a simple yet effective
  model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
  efficiently fine-tunes a series of CLIP models that capture different feature
  spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
  the Feed-Forward Network (FFN). These models can then be transformed into a
  CLIP-MoE with a larger model capacity, leading to significantly enhanced
  performance with minimal computational overhead. To the best of our knowledge,
  Diversified Multiplet Upcycling is the first approach to introduce sparsely
  activated MoE into CLIP foundation models. Extensive experiments demonstrate
  the significant performance of CLIP-MoE across various zero-shot retrieval,
  zero-shot image classification tasks, and downstream Multimodal Large Language
  Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
  Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
  into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
  without requiring further adaptation in downstream frameworks. Through
  Diversified Multiplet Upcycling, we aim to provide valuable insights for future
  research on developing more efficient and effective multimodal learning
  systems.
paper_ai_keywords: Contrastive Language-Image Pre-training (CLIP) information loss coarse-grained features feature spaces Dense pre-trained CLIP checkpoint Feed-Forward Network (FFN) CLIP-MoE model-agnostic strategy Diversified Multiplet Upcycling (DMU) sparsely activated MoE zero-shot retrieval zero-shot image classification Multimodal Large Language Model (MLLM) vision encoder plug-and-play manner