title: Scaling Open-Vocabulary Object Detection
summary: Open-vocabulary object detection has benefited greatly from pretrained
  vision-language models, but is still limited by the amount of available
  detection training data. While detection training data can be expanded by using
  Web image-text pairs as weak supervision, this has not been done at scales
  comparable to image-level pretraining. Here, we scale up detection data with
  self-training, which uses an existing detector to generate pseudo-box
  annotations on image-text pairs. Major challenges in scaling self-training are
  the choice of label space, pseudo-annotation filtering, and training
  efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which
  address these challenges. OWLv2 surpasses the performance of previous
  state-of-the-art open-vocabulary detectors already at comparable training
  scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples,
  yielding further large improvement: With an L/14 architecture, OWL-ST improves
  AP on LVIS rare classes, for which the model has seen no human box annotations,
  from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale
  training for open-world localization, similar to what has been seen for image
  classification and language modelling.
paper_authors._id: 648fbbf6544876b8efd42600
paper_authors.name: Matthias Minderer
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-27T12:55:32.028Z
paper_authors._id: 649ac6e57c36fc2dc6e6b0f4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/649ac6e57c36fc2dc6e6b0f4/Scd3LiYW5Y2qunDDHnPWw.jpeg
paper_authors.fullname: Matthias Minderer
paper_authors.type: user
paper_authors.user: mjlm
paper_authors._id: 648fbbf6544876b8efd42601
paper_authors.name: Alexey Gritsenko
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-19T11:42:43.405Z
paper_authors._id: 62d8f9887b8dc0ba17271415
paper_authors.avatarUrl: /avatars/12ec78d34fd849bad44217b212f31e98.svg
paper_authors.fullname: Alexey Gritsenko
paper_authors.type: user
paper_authors.user: AlexeyG
paper_authors._id: 648fbbf6544876b8efd42602
paper_authors.name: Neil Houlsby
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-07-03T15:23:01.179Z
paper_authors._id: 64a2c7f263f69fb98d3bfdb4
paper_authors.avatarUrl: /avatars/a362a236c0654b7605dcb7673e309335.svg
paper_authors.fullname: Neil Houlsby
paper_authors.type: user
paper_authors.user: neilhoulsby
paper_title: Scaling Open-Vocabulary Object Detection
paper_summary: Open-vocabulary object detection has benefited greatly from pretrained
  vision-language models, but is still limited by the amount of available
  detection training data. While detection training data can be expanded by using
  Web image-text pairs as weak supervision, this has not been done at scales
  comparable to image-level pretraining. Here, we scale up detection data with
  self-training, which uses an existing detector to generate pseudo-box
  annotations on image-text pairs. Major challenges in scaling self-training are
  the choice of label space, pseudo-annotation filtering, and training
  efficiency. We present the OWLv2 model and OWL-ST self-training recipe, which
  address these challenges. OWLv2 surpasses the performance of previous
  state-of-the-art open-vocabulary detectors already at comparable training
  scales (~10M examples). However, with OWL-ST, we can scale to over 1B examples,
  yielding further large improvement: With an L/14 architecture, OWL-ST improves
  AP on LVIS rare classes, for which the model has seen no human box annotations,
  from 31.2% to 44.6% (43% relative improvement). OWL-ST unlocks Web-scale
  training for open-world localization, similar to what has been seen for image
  classification and language modelling.
paper_ai_keywords: open-vocabulary object detection pretrained vision-language models detection training data Web image-text pairs weak supervision self-training pseudo-box annotations OWLv2 model OWL-ST self-training recipe label space pseudo-annotation filtering training efficiency AP (Average Precision) LVIS rare classes human box annotations Web-scale training open-world localization image classification language modelling