title: Building and better understanding vision-language models: insights and
    future directions
summary: The field of vision-language models (VLMs), which take images and texts as
  inputs and output texts, is rapidly evolving and has yet to reach consensus on
  several key aspects of the development pipeline, including data, architecture,
  and training methods. This paper can be seen as a tutorial for building a VLM.
  We begin by providing a comprehensive overview of the current state-of-the-art
  approaches, highlighting the strengths and weaknesses of each, addressing the
  major challenges in the field, and suggesting promising research directions for
  underexplored areas. We then walk through the practical steps to build
  Idefics3-8B, a powerful VLM that significantly outperforms its predecessor
  Idefics2-8B, while being trained efficiently, exclusively on open datasets, and
  using a straightforward pipeline. These steps include the creation of Docmatix,
  a dataset for improving document understanding capabilities, which is 240 times
  larger than previously available datasets. We release the model along with the
  datasets created for its training.
paper_authors._id: 66cc04c301fbc62ecda3f5c2
paper_authors.name: Hugo Laurençon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-28T15:57:23.421Z
paper_authors._id: 6177322d37f32ecb1e2d4cdf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg
paper_authors.fullname: Hugo Laurençon
paper_authors.type: user
paper_authors.user: HugoLaurencon
paper_authors._id: 66cc04c301fbc62ecda3f5c3
paper_authors.name: Andrés Marafioti
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-27T09:22:59.164Z
paper_authors._id: 65d66b494bbd0d92b641cdbb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65d66b494bbd0d92b641cdbb/6-7dm7B-JxcoS1QlCPdMN.jpeg
paper_authors.fullname: Andres Marafioti
paper_authors.type: user
paper_authors.user: andito
paper_authors._id: 66cc04c301fbc62ecda3f5c4
paper_authors.name: Victor Sanh
paper_authors._id: 66cc04c301fbc62ecda3f5c5
paper_authors.name: Léo Tronchon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T13:09:27.214Z
paper_authors._id: 6244866a456803e9500d0f6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652185658647-6244866a456803e9500d0f6a.jpeg
paper_authors.fullname: Leo Tronchon
paper_authors.type: user
paper_authors.user: Leyo
paper_title: Building and better understanding vision-language models: insights and
    future directions
paper_summary: The field of vision-language models (VLMs), which take images and texts as
  inputs and output texts, is rapidly evolving and has yet to reach consensus on
  several key aspects of the development pipeline, including data, architecture,
  and training methods. This paper can be seen as a tutorial for building a VLM.
  We begin by providing a comprehensive overview of the current state-of-the-art
  approaches, highlighting the strengths and weaknesses of each, addressing the
  major challenges in the field, and suggesting promising research directions for
  underexplored areas. We then walk through the practical steps to build
  Idefics3-8B, a powerful VLM that significantly outperforms its predecessor
  Idefics2-8B, while being trained efficiently, exclusively on open datasets, and
  using a straightforward pipeline. These steps include the creation of Docmatix,
  a dataset for improving document understanding capabilities, which is 240 times
  larger than previously available datasets. We release the model along with the
  datasets created for its training.
paper_ai_summary: A comprehensive tutorial on building vision-language models, detailing the development of Idefics3-8B with an improved dataset enhancing document understanding.
paper_ai_keywords: vision-language models VLMs state-of-the-art Docmatix document understanding