title: BRAVE: Broadening the visual encoding of vision-language models
summary: Vision-language models (VLMs) are typically composed of a vision encoder,
  e.g. CLIP, and a language model (LM) that interprets the encoded features to
  solve downstream tasks. Despite remarkable progress, VLMs are subject to
  several shortcomings due to the limited capabilities of vision encoders, e.g.
  "blindness" to certain image features, visual hallucination, etc. To address
  these issues, we study broadening the visual encoding capabilities of VLMs. We
  first comprehensively benchmark several vision encoders with different
  inductive biases for solving VLM tasks. We observe that there is no single
  encoding configuration that consistently achieves top performance across
  different tasks, and encoders with different biases can perform surprisingly
  similarly. Motivated by this, we introduce a method, named BRAVE, that
  consolidates features from multiple frozen encoders into a more versatile
  representation that can be directly fed as the input to a frozen LM. BRAVE
  achieves state-of-the-art performance on a broad range of captioning and VQA
  benchmarks and significantly reduces the aforementioned issues of VLMs, while
  requiring a smaller number of trainable parameters than existing methods and
  having a more compressed representation. Our results highlight the potential of
  incorporating different visual biases for a more broad and contextualized
  visual understanding of VLMs.
paper_authors._id: 661740d051222b41dce25c80
paper_authors.name: Oğuzhan Fatih Kar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-11T14:54:53.857Z
paper_authors._id: 6303d05dd14428368d1866fd
paper_authors.avatarUrl: /avatars/6c86567fac3cb253e0595aa678a2af03.svg
paper_authors.fullname: Oguzhan Fatih Kar
paper_authors.type: user
paper_authors.user: ozo
paper_authors._id: 661740d051222b41dce25c81
paper_authors.name: Alessio Tonioni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T13:27:49.848Z
paper_authors._id: 63402b30e670ff9cf63d8caa
paper_authors.avatarUrl: /avatars/0aee84d132a78d4ec71663836a57a245.svg
paper_authors.fullname: Alessio Tonioni
paper_authors.type: user
paper_authors.user: Alessiot
paper_authors._id: 661740d051222b41dce25c82
paper_authors.name: Petra Poklukar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T13:27:56.294Z
paper_authors._id: 63f271b6f4e30ffd2bd4768b
paper_authors.avatarUrl: /avatars/989c73fac91df3bc16b7a0460b97bc7b.svg
paper_authors.fullname: Petra Poklukar
paper_authors.type: user
paper_authors.user: petrapoklukar
paper_authors._id: 661740d051222b41dce25c83
paper_authors.name: Achin Kulshrestha
paper_authors._id: 661740d051222b41dce25c84
paper_authors.name: Amir Zamir
paper_authors._id: 661740d051222b41dce25c85
paper_authors.name: Federico Tombari
paper_title: BRAVE: Broadening the visual encoding of vision-language models
paper_summary: Vision-language models (VLMs) are typically composed of a vision encoder,
  e.g. CLIP, and a language model (LM) that interprets the encoded features to
  solve downstream tasks. Despite remarkable progress, VLMs are subject to
  several shortcomings due to the limited capabilities of vision encoders, e.g.
  "blindness" to certain image features, visual hallucination, etc. To address
  these issues, we study broadening the visual encoding capabilities of VLMs. We
  first comprehensively benchmark several vision encoders with different
  inductive biases for solving VLM tasks. We observe that there is no single
  encoding configuration that consistently achieves top performance across
  different tasks, and encoders with different biases can perform surprisingly
  similarly. Motivated by this, we introduce a method, named BRAVE, that
  consolidates features from multiple frozen encoders into a more versatile
  representation that can be directly fed as the input to a frozen LM. BRAVE
  achieves state-of-the-art performance on a broad range of captioning and VQA
  benchmarks and significantly reduces the aforementioned issues of VLMs, while
  requiring a smaller number of trainable parameters than existing methods and
  having a more compressed representation. Our results highlight the potential of
  incorporating different visual biases for a more broad and contextualized
  visual understanding of VLMs.
paper_ai_keywords: Vision-language models (VLMs) CLIP language model (LM) visual encoding capabilities inductive biases broadening benchmarking BRAVE consolidated features versatile representation captioning VQA (Visual Question Answering) benchmarks visual hallucination visual understanding