Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset
<|reference_start|>Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset: Using vision-language models (VLMs) in web development presents a promising strategy to increase efficiency and unblock no-code solutions: by providing a screenshot or a sketch of a UI, a VLM could generate the code to reproduce it, for instance in a language like HTML. Despite the advancements in VLMs for various tasks, the specific challenge of converting a screenshot into a corresponding HTML has been minimally explored. We posit that this is mainly due to the absence of a suitable, high-quality dataset. This work introduces WebSight, a synthetic dataset consisting of 2 million pairs of HTML codes and their corresponding screenshots. We fine-tune a foundational VLM on our dataset and show proficiency in converting webpage screenshots to functional HTML code. To accelerate the research in this area, we open-source WebSight.<|reference_end|>
@article{laurençon2024unlocking,
    title={Unlocking the conversion of Web Screenshots into HTML Code with the
  WebSight Dataset},
    author={Hugo Laurenc{c}on, L'eo Tronchon, Victor Sanh},
    journal={arXiv preprint arXiv:2403.09029},
    year={2024},
    archivePrefix={arXiv},
    eprint={2403.09029},
    primaryClass={cs.HC cs.AI cs.CV}
}