title: FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
summary: Contrastive language-image pre-training aligns the features of text-image
  pairs in a common latent space via distinct encoders for each modality. While
  this approach achieves impressive performance in several zero-shot tasks, it
  cannot natively handle multimodal inputs, i.e., encoding image and text into a
  single feature vector. As a remedy, it is common practice to use additional
  modules to merge the features extracted by the unimodal encoders. In this work,
  we present FuseLIP, an alternative architecture for multimodal embedding.
  Leveraging recent progress in discrete image tokenizers, we propose to use a
  single transformer model which operates on an extended vocabulary of text and
  image tokens. This early fusion approach allows the different modalities to
  interact at each depth of encoding and obtain richer representations compared
  to common late fusion. We collect new datasets for multimodal pre-training and
  evaluation, designing challenging tasks for multimodal encoder models. We show
  that FuseLIP outperforms other approaches in multimodal embedding tasks such as
  VQA and text-guided image transformation retrieval, while being comparable to
  baselines on unimodal tasks.
paper_authors._id: 684002718bd5bff9918ce018
paper_authors.name: Christian Schlarmann
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-04T08:53:34.919Z
paper_authors._id: 6310a6bb0a43f97f6c5567d3
paper_authors.avatarUrl: /avatars/04b07c3a6c811337939c951567cb2bf2.svg
paper_authors.fullname: Christian Schlarmann
paper_authors.type: user
paper_authors.user: chs20
paper_authors._id: 684002718bd5bff9918ce019
paper_authors.name: Francesco Croce
paper_authors._id: 684002718bd5bff9918ce01a
paper_authors.name: Nicolas Flammarion
paper_authors._id: 684002718bd5bff9918ce01b
paper_authors.name: Matthias Hein
paper_title: FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
paper_summary: Contrastive language-image pre-training aligns the features of text-image
  pairs in a common latent space via distinct encoders for each modality. While
  this approach achieves impressive performance in several zero-shot tasks, it
  cannot natively handle multimodal inputs, i.e., encoding image and text into a
  single feature vector. As a remedy, it is common practice to use additional
  modules to merge the features extracted by the unimodal encoders. In this work,
  we present FuseLIP, an alternative architecture for multimodal embedding.
  Leveraging recent progress in discrete image tokenizers, we propose to use a
  single transformer model which operates on an extended vocabulary of text and
  image tokens. This early fusion approach allows the different modalities to
  interact at each depth of encoding and obtain richer representations compared
  to common late fusion. We collect new datasets for multimodal pre-training and
  evaluation, designing challenging tasks for multimodal encoder models. We show
  that FuseLIP outperforms other approaches in multimodal embedding tasks such as
  VQA and text-guided image transformation retrieval, while being comparable to
  baselines on unimodal tasks.
paper_ai_summary: FuseLIP is a transformer-based architecture that uses a shared vocabulary for text and image tokens to enhance multimodal embedding and outperforms existing models in tasks such as VQA and text-guided image retrieval.
paper_ai_keywords: contrastive language-image pre-training latent space unimodal encoders multimodal inputs feature vector late fusion early fusion transformer model discrete image tokenizers multimodal pre-training VQA text-guided image transformation retrieval