title: Enhancing Vision-Language Pre-training with Rich Supervisions
summary: We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
  pre-training paradigm for Vision-Language Models using data from large-scale
  web screenshot rendering. Using web screenshots unlocks a treasure trove of
  visual and textual cues that are not present in using image-text pairs. In S4,
  we leverage the inherent tree-structured hierarchy of HTML elements and the
  spatial localization to carefully design 10 pre-training tasks with large scale
  annotated data. These tasks resemble downstream tasks across different domains
  and the annotations are cheap to obtain. We demonstrate that, compared to
  current screenshot pre-training objectives, our innovative pre-training method
  significantly enhances performance of image-to-text model in nine varied and
  popular downstream tasks - up to 76.1% improvements on Table Detection, and at
  least 1% on Widget Captioning.
paper_authors._id: 65e93d92c331a092145618f8
paper_authors.name: Yuan Gao
paper_authors._id: 65e93d92c331a092145618f9
paper_authors.name: Kunyu Shi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-07T16:46:06.124Z
paper_authors._id: 65e9ea877c911508683a27a2
paper_authors.avatarUrl: /avatars/eb0a6155a6bdce7d62d9b04cc9fedafa.svg
paper_authors.fullname: Kunyu Shi
paper_authors.type: user
paper_authors.user: shikunyu8
paper_authors._id: 65e93d92c331a092145618fa
paper_authors.name: Pengkai Zhu
paper_authors._id: 65e93d92c331a092145618fb
paper_authors.name: Edouard Belval
paper_authors._id: 65e93d92c331a092145618fc
paper_authors.name: Oren Nuriel
paper_authors._id: 65e93d92c331a092145618fd
paper_authors.name: Srikar Appalaraju
paper_authors._id: 65e93d92c331a092145618fe
paper_authors.name: Shabnam Ghadar
paper_authors._id: 65e93d92c331a092145618ff
paper_authors.name: Vijay Mahadevan
paper_authors._id: 65e93d92c331a09214561900
paper_authors.name: Zhuowen Tu
paper_authors._id: 65e93d92c331a09214561901
paper_authors.name: Stefano Soatto
paper_title: Enhancing Vision-Language Pre-training with Rich Supervisions
paper_summary: We propose Strongly Supervised pre-training with ScreenShots (S4) - a novel
  pre-training paradigm for Vision-Language Models using data from large-scale
  web screenshot rendering. Using web screenshots unlocks a treasure trove of
  visual and textual cues that are not present in using image-text pairs. In S4,
  we leverage the inherent tree-structured hierarchy of HTML elements and the
  spatial localization to carefully design 10 pre-training tasks with large scale
  annotated data. These tasks resemble downstream tasks across different domains
  and the annotations are cheap to obtain. We demonstrate that, compared to
  current screenshot pre-training objectives, our innovative pre-training method
  significantly enhances performance of image-to-text model in nine varied and
  popular downstream tasks - up to 76.1% improvements on Table Detection, and at
  least 1% on Widget Captioning.
paper_ai_summary: Strongly Supervised pre-training with ScreenShots (S4) enhances Vision-Language Models by using web screenshots with tree-structured HTML elements and spatial localization, resulting in significant improvements in diverse downstream tasks.
paper_ai_keywords: Strongly Supervised pre-training ScreenShots Vision-Language Models web screenshots tree-structured hierarchy HTML elements spatial localization pre-training tasks image-to-text model Table Detection Widget Captioning