title: ScreenAI: A Vision-Language Model for UI and Infographics Understanding
summary: Screen user interfaces (UIs) and infographics, sharing similar visual
  language and design principles, play important roles in human communication and
  human-machine interaction. We introduce ScreenAI, a vision-language model that
  specializes in UI and infographics understanding. Our model improves upon the
  PaLI architecture with the flexible patching strategy of pix2struct and is
  trained on a unique mixture of datasets. At the heart of this mixture is a
  novel screen annotation task in which the model has to identify the type and
  location of UI elements. We use these text annotations to describe screens to
  Large Language Models and automatically generate question-answering (QA), UI
  navigation, and summarization training datasets at scale. We run ablation
  studies to demonstrate the impact of these design choices. At only 5B
  parameters, ScreenAI achieves new state-of-the-artresults on UI- and
  infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
  Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
  InfographicVQA) compared to models of similar size. Finally, we release three
  new datasets: one focused on the screen annotation task and two others focused
  on question answering.
paper_authors._id: 65c43d0847a0ffffa9680446
paper_authors.name: Gilles Baechler
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:54:45.163Z
paper_authors._id: 633bf74e9a0fb78266be26cf
paper_authors.avatarUrl: /avatars/360e37dff6841ee97779743e94e62213.svg
paper_authors.fullname: Gilles Baechler
paper_authors.type: user
paper_authors.user: g-les
paper_authors._id: 65c43d0847a0ffffa9680447
paper_authors.name: Srinivas Sunkara
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:54:51.422Z
paper_authors._id: 65658dceee904e90dc3c73e2
paper_authors.avatarUrl: /avatars/71da82fec0b1de30be0c1c93ca177723.svg
paper_authors.fullname: Srinivas Sunkara
paper_authors.type: user
paper_authors.user: embraceai
paper_authors._id: 65c43d0847a0ffffa9680448
paper_authors.name: Maria Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:54:58.487Z
paper_authors._id: 6325cadb2858ff914be9bc2e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663421120944-noauth.png
paper_authors.fullname: Maria Wang
paper_authors.type: user
paper_authors.user: MariaW
paper_authors._id: 65c43d0847a0ffffa9680449
paper_authors.name: Fedir Zubach
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:55:04.893Z
paper_authors._id: 65364efd5f5e918bdc26c64c
paper_authors.avatarUrl: /avatars/0b6c92c98e0dfc0ef437bb781dd51da5.svg
paper_authors.fullname: Fedir Zubach
paper_authors.type: user
paper_authors.user: fzubach
paper_authors._id: 65c43d0847a0ffffa968044a
paper_authors.name: Hassan Mansoor
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:55:10.776Z
paper_authors._id: 629098fd5463575364e7697a
paper_authors.avatarUrl: /avatars/e45f3ee28bff3571f76caf847d3c36db.svg
paper_authors.fullname: Hassan Mansoor
paper_authors.type: user
paper_authors.user: HassanMansoor
paper_authors._id: 65c43d0847a0ffffa968044b
paper_authors.name: Vincent Etter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:55:16.541Z
paper_authors._id: 6336dd00aeab734eec3f82f3
paper_authors.avatarUrl: /avatars/460ef0ede333e4cacf97afd1f263e612.svg
paper_authors.fullname: Vincent Etter
paper_authors.type: user
paper_authors.user: w00ka1
paper_authors._id: 65c43d0847a0ffffa968044c
paper_authors.name: Victor Cărbune
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:55:30.188Z
paper_authors._id: 656054efc40c3a6a0d9d7b3e
paper_authors.avatarUrl: /avatars/6d93e933df1cd59b836f5f79b5e938bf.svg
paper_authors.fullname: Victor Carbune
paper_authors.type: user
paper_authors.user: vcarbune
paper_authors._id: 65c43d0847a0ffffa968044d
paper_authors.name: Jason Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-09T08:05:54.738Z
paper_authors._id: 635f700b398ff343c4fd5c3b
paper_authors.avatarUrl: /avatars/d62ef6809b9d8da8713ecdafa36fb137.svg
paper_authors.fullname: Jason Lin
paper_authors.type: user
paper_authors.user: jaxball
paper_authors._id: 65c43d0847a0ffffa968044e
paper_authors.name: Jindong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-08T11:55:52.413Z
paper_authors._id: 635242b8b3678a43742baec6
paper_authors.avatarUrl: /avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
paper_authors.fullname: chenjindong
paper_authors.type: user
paper_authors.user: chenjindong
paper_authors._id: 65c43d0847a0ffffa968044f
paper_authors.name: Abhanshu Sharma
paper_title: ScreenAI: A Vision-Language Model for UI and Infographics Understanding
paper_summary: Screen user interfaces (UIs) and infographics, sharing similar visual
  language and design principles, play important roles in human communication and
  human-machine interaction. We introduce ScreenAI, a vision-language model that
  specializes in UI and infographics understanding. Our model improves upon the
  PaLI architecture with the flexible patching strategy of pix2struct and is
  trained on a unique mixture of datasets. At the heart of this mixture is a
  novel screen annotation task in which the model has to identify the type and
  location of UI elements. We use these text annotations to describe screens to
  Large Language Models and automatically generate question-answering (QA), UI
  navigation, and summarization training datasets at scale. We run ablation
  studies to demonstrate the impact of these design choices. At only 5B
  parameters, ScreenAI achieves new state-of-the-artresults on UI- and
  infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
  Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
  InfographicVQA) compared to models of similar size. Finally, we release three
  new datasets: one focused on the screen annotation task and two others focused
  on question answering.
paper_ai_keywords: vision-language model UIF architecture flexible patching strategy pix2struct screen annotation task Large Language Models question-answering (QA) UI navigation summarization ablation studies Multi-page DocVQA WebSRC MoTIF Widget Captioning Chart QA DocVQA InfographicVQA