What matters when building vision-language models?
The growing interest in vision-language models (VLMs) has been driven by
improvements in large language models and vision transformers. Despite the
abundance of literature on this subject, we observe that critical decisions
regarding the design of VLMs are often not justified. We argue that these
unsupported decisions impede progress in the field by making it difficult to
identify which choices improve model performance. To address this issue, we
conduct extensive experiments around pre-trained models, architecture choice,
data, and training methods. Our consolidation of findings includes the
development of Idefics2, an efficient foundational VLM of 8 billion parameters.
Idefics2 achieves state-of-the-art performance within its size category across
various multimodal benchmarks, and is often on par with models four times its
size. We release the model (base, instructed, and chat) along with the datasets
created for its training.
6638c1a5d26a707999bbac28
Hugo Laurençon
claimed_verified
2024-05-13T07:48:59.560Z
6177322d37f32ecb1e2d4cdf
https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg
Hugo Laurençon
user
HugoLaurencon
6638c1a5d26a707999bbac29
Léo Tronchon
claimed_verified
2024-05-07T08:33:18.440Z
6244866a456803e9500d0f6a
https://cdn-avatars.huggingface.co/v1/production/uploads/1652185658647-6244866a456803e9500d0f6a.jpeg
Leo Tronchon
user
Leyo
6638c1a5d26a707999bbac2a
Matthieu Cord
6638c1a5d26a707999bbac2b
Victor Sanh
claimed_verified
2024-05-06T13:15:17.754Z
5ecea265968f6028e0559fa5
https://cdn-avatars.huggingface.co/v1/production/uploads/1619623771844-5ecea265968f6028e0559fa5.jpeg
Victor Sanh
user
VictorSanh
What matters when building vision-language models?
The growing interest in vision-language models (VLMs) has been driven by
improvements in large language models and vision transformers. Despite the
abundance of literature on this subject, we observe that critical decisions
regarding the design of VLMs are often not justified. We argue that these
unsupported decisions impede progress in the field by making it difficult to
identify which choices improve model performance. To address this issue, we
conduct extensive experiments around pre-trained models, architecture choice,
data, and training methods. Our consolidation of findings includes the
development of Idefics2, an efficient foundational VLM of 8 billion parameters.
Idefics2 achieves state-of-the-art performance within its size category across
various multimodal benchmarks, and is often on par with models four times its
size. We release the model (base, instructed, and chat) along with the datasets
created for its training.
Idefics2, a vision-language model with 8 billion parameters, achieves state-of-the-art performance on multimodal benchmarks through extensive experimental validation.
vision-language models large language models vision transformers pre-trained models architecture choice Idefics2 multimodal benchmarks