title: Multi-Modal Classifiers for Open-Vocabulary Object Detection
summary: The goal of this paper is open-vocabulary object detection (OVOD)
  x2013 building a model that can detect objects beyond the set of
  categories seen at training, thus enabling the user to specify categories of
  interest at inference without the need for model retraining. We adopt a
  standard two-stage object detector architecture, and explore three ways for
  specifying novel categories: via language descriptions, via image exemplars, or
  via a combination of the two. We make three contributions: first, we prompt a
  large language model (LLM) to generate informative language descriptions for
  object classes, and construct powerful text-based classifiers; second, we
  employ a visual aggregator on image exemplars that can ingest any number of
  images as input, forming vision-based classifiers; and third, we provide a
  simple method to fuse information from language descriptions and image
  exemplars, yielding a multi-modal classifier. When evaluating on the
  challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our
  text-based classifiers outperform all previous OVOD works; (ii) our
  vision-based classifiers perform as well as text-based classifiers in prior
  work; (iii) using multi-modal classifiers perform better than either modality
  alone; and finally, (iv) our text-based and multi-modal classifiers yield
  better performance than a fully-supervised detector.
paper_authors._id: 648682986a2a8eca9d1ea69c
paper_authors.name: Prannay Kaul
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-12T18:59:18.187Z
paper_authors._id: 63500f69eaf46b3c3d893c1c
paper_authors.avatarUrl: /avatars/119f33d862bdacc9d6e45eed213eb5a7.svg
paper_authors.fullname: Prannay Kaul
paper_authors.type: user
paper_authors.user: prannaykaul
paper_authors._id: 648682986a2a8eca9d1ea69d
paper_authors.name: Weidi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-12T10:04:01.923Z
paper_authors._id: 63f1841f2f7c0152e8748ec4
paper_authors.avatarUrl: /avatars/5398658c7d0cf556531a625a4ca5d18a.svg
paper_authors.fullname: Xie
paper_authors.type: user
paper_authors.user: Weidi
paper_authors._id: 648682986a2a8eca9d1ea69e
paper_authors.name: Andrew Zisserman
paper_title: Multi-Modal Classifiers for Open-Vocabulary Object Detection
paper_summary: The goal of this paper is open-vocabulary object detection (OVOD)
  x2013 building a model that can detect objects beyond the set of
  categories seen at training, thus enabling the user to specify categories of
  interest at inference without the need for model retraining. We adopt a
  standard two-stage object detector architecture, and explore three ways for
  specifying novel categories: via language descriptions, via image exemplars, or
  via a combination of the two. We make three contributions: first, we prompt a
  large language model (LLM) to generate informative language descriptions for
  object classes, and construct powerful text-based classifiers; second, we
  employ a visual aggregator on image exemplars that can ingest any number of
  images as input, forming vision-based classifiers; and third, we provide a
  simple method to fuse information from language descriptions and image
  exemplars, yielding a multi-modal classifier. When evaluating on the
  challenging LVIS open-vocabulary benchmark we demonstrate that: (i) our
  text-based classifiers outperform all previous OVOD works; (ii) our
  vision-based classifiers perform as well as text-based classifiers in prior
  work; (iii) using multi-modal classifiers perform better than either modality
  alone; and finally, (iv) our text-based and multi-modal classifiers yield
  better performance than a fully-supervised detector.
paper_ai_keywords: open-vocabulary object detection (OVOD) large language model (LLM) text-based classifiers visual aggregator vision-based classifiers multi-modal classifier LVIS open-vocabulary benchmark