title: Unlocking the conversion of Web Screenshots into HTML Code with the
    WebSight Dataset
summary: Using vision-language models (VLMs) in web development presents a promising
  strategy to increase efficiency and unblock no-code solutions: by providing a
  screenshot or a sketch of a UI, a VLM could generate the code to reproduce it,
  for instance in a language like HTML. Despite the advancements in VLMs for
  various tasks, the specific challenge of converting a screenshot into a
  corresponding HTML has been minimally explored. We posit that this is mainly
  due to the absence of a suitable, high-quality dataset. This work introduces
  WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and
  their corresponding screenshots. We fine-tune a foundational VLM on our dataset
  and show proficiency in converting webpage screenshots to functional HTML code.
  To accelerate the research in this area, we open-source WebSight.
paper_authors._id: 65f3a30e2b4e85e2e8898b2d
paper_authors.name: Hugo Laurençon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:42:49.912Z
paper_authors._id: 6177322d37f32ecb1e2d4cdf
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1635201569275-noauth.jpeg
paper_authors.fullname: Hugo Laurençon
paper_authors.type: user
paper_authors.user: HugoLaurencon
paper_authors._id: 65f3a30e2b4e85e2e8898b2e
paper_authors.name: Léo Tronchon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:43:01.647Z
paper_authors._id: 6244866a456803e9500d0f6a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652185658647-6244866a456803e9500d0f6a.jpeg
paper_authors.fullname: Leo Tronchon
paper_authors.type: user
paper_authors.user: Leyo
paper_authors._id: 65f3a30e2b4e85e2e8898b2f
paper_authors.name: Victor Sanh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T08:43:09.692Z
paper_authors._id: 5ecea265968f6028e0559fa5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1619623771844-5ecea265968f6028e0559fa5.jpeg
paper_authors.fullname: Victor Sanh
paper_authors.type: user
paper_authors.user: VictorSanh
paper_title: Unlocking the conversion of Web Screenshots into HTML Code with the
    WebSight Dataset
paper_summary: Using vision-language models (VLMs) in web development presents a promising
  strategy to increase efficiency and unblock no-code solutions: by providing a
  screenshot or a sketch of a UI, a VLM could generate the code to reproduce it,
  for instance in a language like HTML. Despite the advancements in VLMs for
  various tasks, the specific challenge of converting a screenshot into a
  corresponding HTML has been minimally explored. We posit that this is mainly
  due to the absence of a suitable, high-quality dataset. This work introduces
  WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and
  their corresponding screenshots. We fine-tune a foundational VLM on our dataset
  and show proficiency in converting webpage screenshots to functional HTML code.
  To accelerate the research in this area, we open-source WebSight.
paper_ai_summary: A synthetic dataset of HTML code and screenshots is introduced to enhance the ability of vision-language models to convert screenshots into HTML code.
paper_ai_keywords: vision-language models VLMs WebSight synthetic dataset HTML codes screenshots fine-tune