title: Region-Aware Pretraining for Open-Vocabulary Object Detection with
    Vision Transformers
summary: We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a
  contrastive image-text pretraining recipe to bridge the gap between image-level
  pretraining and open-vocabulary object detection. At the pretraining phase, we
  propose to randomly crop and resize regions of positional embeddings instead of
  using the whole image positional embeddings. This better matches the use of
  positional embeddings at region-level in the detection finetuning phase. In
  addition, we replace the common softmax cross entropy loss in contrastive
  learning with focal loss to better learn the informative yet difficult
  examples. Finally, we leverage recent advances in novel object proposals to
  improve open-vocabulary detection finetuning. We evaluate our full model on the
  LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer.
  RO-ViT achieves a state-of-the-art 32.1 AP_r on LVIS, surpassing the best
  existing approach by +5.8 points in addition to competitive zero-shot transfer
  detection. Surprisingly, RO-ViT improves the image-level representation as well
  and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr
  image-text retrieval benchmarks, outperforming competitive approaches with
  larger models.
paper_authors._id: 645dc4767c7258d904e97553
paper_authors.name: Dahun Kim
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-15T07:34:03.283Z
paper_authors._id: 6461d65b5ea1bc9a9da98b88
paper_authors.avatarUrl: /avatars/fd6e3715ca8f918ca074356244bb9ce9.svg
paper_authors.fullname: Dahun Kim
paper_authors.type: user
paper_authors.user: mcahny
paper_authors._id: 645dc4767c7258d904e97554
paper_authors.name: Anelia Angelova
paper_authors._id: 645dc4767c7258d904e97555
paper_authors.name: Weicheng Kuo
paper_title: Region-Aware Pretraining for Open-Vocabulary Object Detection with
    Vision Transformers
paper_summary: We present Region-aware Open-vocabulary Vision Transformers (RO-ViT) - a
  contrastive image-text pretraining recipe to bridge the gap between image-level
  pretraining and open-vocabulary object detection. At the pretraining phase, we
  propose to randomly crop and resize regions of positional embeddings instead of
  using the whole image positional embeddings. This better matches the use of
  positional embeddings at region-level in the detection finetuning phase. In
  addition, we replace the common softmax cross entropy loss in contrastive
  learning with focal loss to better learn the informative yet difficult
  examples. Finally, we leverage recent advances in novel object proposals to
  improve open-vocabulary detection finetuning. We evaluate our full model on the
  LVIS and COCO open-vocabulary detection benchmarks and zero-shot transfer.
  RO-ViT achieves a state-of-the-art 32.1 AP_r on LVIS, surpassing the best
  existing approach by +5.8 points in addition to competitive zero-shot transfer
  detection. Surprisingly, RO-ViT improves the image-level representation as well
  and achieves the state of the art on 9 out of 12 metrics on COCO and Flickr
  image-text retrieval benchmarks, outperforming competitive approaches with
  larger models.
paper_ai_keywords: Region-aware Open-vocabulary Vision Transformers (RO-ViT) multimodal image-level pretraining open-vocabulary object detection random crop and resize positional embeddings contrastive learning softmax cross entropy loss focal loss informative examples difficult examples novel object proposals open-vocabulary detection LVIS benchmark COCO benchmark zero-shot transfer $AP_r$ image-level representation gold descriptions image-text retrieval benchmarks