CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
  Multiplet Upcycling
In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
cornerstone in multimodal intelligence. However, recent studies have identified
that the information loss in the CLIP encoding process is substantial, and CLIP
tends to capture only coarse-grained features from the input. This deficiency
significantly limits the ability of a single CLIP model to handle images rich
in visual detail. In this work, we propose a simple yet effective
model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
efficiently fine-tunes a series of CLIP models that capture different feature
spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
the Feed-Forward Network (FFN). These models can then be transformed into a
CLIP-MoE with a larger model capacity, leading to significantly enhanced
performance with minimal computational overhead. To the best of our knowledge,
Diversified Multiplet Upcycling is the first approach to introduce sparsely
activated MoE into CLIP foundation models. Extensive experiments demonstrate
the significant performance of CLIP-MoE across various zero-shot retrieval,
zero-shot image classification tasks, and downstream Multimodal Large Language
Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
without requiring further adaptation in downstream frameworks. Through
Diversified Multiplet Upcycling, we aim to provide valuable insights for future
research on developing more efficient and effective multimodal learning
systems.
66ffab22dbdb4b0b9f759af9
Jihai Zhang
66ffab22dbdb4b0b9f759afa
Xiaoye Qu
claimed_verified
2024-10-04T08:46:35.159Z
64cb54da1af278541d663708
/avatars/c44507cc92bb2e83154bad31b90ce6dd.svg
Xiaoye Qu
user
Xiaoye08
66ffab22dbdb4b0b9f759afb
Tong Zhu
claimed_verified
2024-10-04T09:02:32.963Z
629454301ae2138079f7ff31
https://cdn-avatars.huggingface.co/v1/production/uploads/629454301ae2138079f7ff31/rVtbF-j06gDiYzomTeVTc.jpeg
Tong Zhu
user
Spico
66ffab22dbdb4b0b9f759afc
Yu Cheng
claimed_verified
2024-10-09T16:26:29.239Z
67017abfe4d49b157ac534d9
/avatars/997e1b9f54b27a7728a9d4abfee4ba91.svg
Yu Cheng
user
ych133
CLIP-MoE: Towards Building Mixture of Experts for CLIP with Diversified
  Multiplet Upcycling
In recent years, Contrastive Language-Image Pre-training (CLIP) has become a
cornerstone in multimodal intelligence. However, recent studies have identified
that the information loss in the CLIP encoding process is substantial, and CLIP
tends to capture only coarse-grained features from the input. This deficiency
significantly limits the ability of a single CLIP model to handle images rich
in visual detail. In this work, we propose a simple yet effective
model-agnostic strategy, Diversified Multiplet Upcycling (DMU), for CLIP. DMU
efficiently fine-tunes a series of CLIP models that capture different feature
spaces, from a dense pre-trained CLIP checkpoint, sharing parameters except for
the Feed-Forward Network (FFN). These models can then be transformed into a
CLIP-MoE with a larger model capacity, leading to significantly enhanced
performance with minimal computational overhead. To the best of our knowledge,
Diversified Multiplet Upcycling is the first approach to introduce sparsely
activated MoE into CLIP foundation models. Extensive experiments demonstrate
the significant performance of CLIP-MoE across various zero-shot retrieval,
zero-shot image classification tasks, and downstream Multimodal Large Language
Model (MLLM) benchmarks by serving as a vision encoder. Furthermore,
Diversified Multiplet Upcycling enables the conversion of any dense CLIP model
into CLIP-MoEs, which can seamlessly replace CLIP in a plug-and-play manner
without requiring further adaptation in downstream frameworks. Through
Diversified Multiplet Upcycling, we aim to provide valuable insights for future
research on developing more efficient and effective multimodal learning
systems.
Diversified Multiplet Upcycling (DMU) enhances CLIP models by fine-tuning them to capture diverse feature spaces, transforming them into CLIP-MoEs, and demonstrating superior performance in various zero-shot tasks and MLLM benchmarks.
Contrastive Language-Image Pre-training CLIP Diversified Multiplet Upcycling DMU Feature spaces Feed-Forward Network FFN CLIP-MoE zero-shot retrieval zero-shot image classification Multimodal Large Language Model MLLM vision encoder