title: Contrastive Feature Masking Open-Vocabulary Vision Transformer
summary: We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an
  image-text pretraining methodology that achieves simultaneous learning of
  image- and region-level representation for open-vocabulary object detection
  (OVD). Our approach combines the masked autoencoder (MAE) objective into the
  contrastive learning objective to improve the representation for localization
  tasks. Unlike standard MAE, we perform reconstruction in the joint image-text
  embedding space, rather than the pixel space as is customary with the classical
  MAE method, which causes the model to better learn region-level semantics.
  Moreover, we introduce Positional Embedding Dropout (PED) to address scale
  variation between image-text pretraining and detection finetuning by randomly
  dropping out the positional embeddings during pretraining. PED improves
  detection performance and enables the use of a frozen ViT backbone as a region
  classifier, preventing the forgetting of open-vocabulary knowledge during
  detection finetuning. On LVIS open-vocabulary detection benchmark, CFM-ViT
  achieves a state-of-the-art 33.9 APr, surpassing the best approach by 7.6
  points and achieves better zero-shot detection transfer. Finally, CFM-ViT
  acquires strong image-level representation, outperforming the state of the art
  on 8 out of 12 metrics on zero-shot image-text retrieval benchmarks.
paper_authors._id: 64f83102a9b6fed18c6bf012
paper_authors.name: Dahun Kim
paper_authors._id: 64f83102a9b6fed18c6bf013
paper_authors.name: Anelia Angelova
paper_authors._id: 64f83102a9b6fed18c6bf014
paper_authors.name: Weicheng Kuo
paper_title: Contrastive Feature Masking Open-Vocabulary Vision Transformer
paper_summary: We present Contrastive Feature Masking Vision Transformer (CFM-ViT) - an
  image-text pretraining methodology that achieves simultaneous learning of
  image- and region-level representation for open-vocabulary object detection
  (OVD). Our approach combines the masked autoencoder (MAE) objective into the
  contrastive learning objective to improve the representation for localization
  tasks. Unlike standard MAE, we perform reconstruction in the joint image-text
  embedding space, rather than the pixel space as is customary with the classical
  MAE method, which causes the model to better learn region-level semantics.
  Moreover, we introduce Positional Embedding Dropout (PED) to address scale
  variation between image-text pretraining and detection finetuning by randomly
  dropping out the positional embeddings during pretraining. PED improves
  detection performance and enables the use of a frozen ViT backbone as a region
  classifier, preventing the forgetting of open-vocabulary knowledge during
  detection finetuning. On LVIS open-vocabulary detection benchmark, CFM-ViT
  achieves a state-of-the-art 33.9 APr, surpassing the best approach by 7.6
  points and achieves better zero-shot detection transfer. Finally, CFM-ViT
  acquires strong image-level representation, outperforming the state of the art
  on 8 out of 12 metrics on zero-shot image-text retrieval benchmarks.
paper_ai_summary: CFM-ViT enhances open-vocabulary object detection and image-text retrieval through combined masked and contrastive learning objectives, positional embedding dropout, and region-level representation learning.
paper_ai_keywords: Contrastive Feature Masking Vision Transformer CFM-ViT masked autoencoder contrastive learning joint image-text embedding space pixel space Positional Embedding Dropout PED open-vocabulary object detection OVD LVIS detection benchmark zero-shot detection zero-shot image-text retrieval