title: What matters when building vision-language models?
summary: The growing interest in vision-language models (VLMs) has been driven by
  improvements in large language models and vision transformers. Despite the
  abundance of literature on this subject, we observe that critical decisions
  regarding the design of VLMs are often not justified. We argue that these
  unsupported decisions impede progress in the field by making it difficult to
  identify which choices improve model performance. To address this issue, we
  conduct extensive experiments around pre-trained models, architecture choice,
  data, and training methods. Our consolidation of findings includes the
  development of Idefics2, an efficient foundational VLM of 8 billion parameters.
  Idefics2 achieves state-of-the-art performance within its size category across
  various multimodal benchmarks, and is often on par with models four times its
  size. We release the model (base, instructed, and chat) along with the datasets
  created for its training.
paper_authors._id: 6638c1a5d26a707999bbac28
paper_authors.name: Hugo Laurençon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-13T07:48:59.560Z
paper_authors._id: 6177322d37f32ecb1e2d4cdf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg
paper_authors.fullname: Hugo Laurençon
paper_authors.type: user
paper_authors.user: HugoLaurencon
paper_authors._id: 6638c1a5d26a707999bbac29
paper_authors.name: Léo Tronchon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-07T08:33:18.440Z
paper_authors._id: 6244866a456803e9500d0f6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652185658647-6244866a456803e9500d0f6a.jpeg
paper_authors.fullname: Leo Tronchon
paper_authors.type: user
paper_authors.user: Leyo
paper_authors._id: 6638c1a5d26a707999bbac2a
paper_authors.name: Matthieu Cord
paper_authors._id: 6638c1a5d26a707999bbac2b
paper_authors.name: Victor Sanh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-06T13:15:17.754Z
paper_authors._id: 5ecea265968f6028e0559fa5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619623771844-5ecea265968f6028e0559fa5.jpeg
paper_authors.fullname: Victor Sanh
paper_authors.type: user
paper_authors.user: VictorSanh
paper_title: What matters when building vision-language models?
paper_summary: The growing interest in vision-language models (VLMs) has been driven by
  improvements in large language models and vision transformers. Despite the
  abundance of literature on this subject, we observe that critical decisions
  regarding the design of VLMs are often not justified. We argue that these
  unsupported decisions impede progress in the field by making it difficult to
  identify which choices improve model performance. To address this issue, we
  conduct extensive experiments around pre-trained models, architecture choice,
  data, and training methods. Our consolidation of findings includes the
  development of Idefics2, an efficient foundational VLM of 8 billion parameters.
  Idefics2 achieves state-of-the-art performance within its size category across
  various multimodal benchmarks, and is often on par with models four times its
  size. We release the model (base, instructed, and chat) along with the datasets
  created for its training.
paper_ai_summary: Idefics2, a vision-language model with 8 billion parameters, achieves state-of-the-art performance on multimodal benchmarks through extensive experimental validation.
paper_ai_keywords: vision-language models large language models vision transformers pre-trained models architecture choice Idefics2 multimodal benchmarks