ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Screen user interfaces (UIs) and infographics, sharing similar visual
language and design principles, play important roles in human communication and
human-machine interaction. We introduce ScreenAI, a vision-language model that
specializes in UI and infographics understanding. Our model improves upon the
PaLI architecture with the flexible patching strategy of pix2struct and is
trained on a unique mixture of datasets. At the heart of this mixture is a
novel screen annotation task in which the model has to identify the type and
location of UI elements. We use these text annotations to describe screens to
Large Language Models and automatically generate question-answering (QA), UI
navigation, and summarization training datasets at scale. We run ablation
studies to demonstrate the impact of these design choices. At only 5B
parameters, ScreenAI achieves new state-of-the-artresults on UI- and
infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
InfographicVQA) compared to models of similar size. Finally, we release three
new datasets: one focused on the screen annotation task and two others focused
on question answering.
65c43d0847a0ffffa9680446
Gilles Baechler
admin_assigned
2024-02-08T11:54:45.163Z
633bf74e9a0fb78266be26cf
/avatars/360e37dff6841ee97779743e94e62213.svg
Gilles Baechler
user
g-les
65c43d0847a0ffffa9680447
Srinivas Sunkara
admin_assigned
2024-02-08T11:54:51.422Z
65658dceee904e90dc3c73e2
/avatars/71da82fec0b1de30be0c1c93ca177723.svg
Srinivas Sunkara
user
embraceai
65c43d0847a0ffffa9680448
Maria Wang
admin_assigned
2024-02-08T11:54:58.487Z
6325cadb2858ff914be9bc2e
https://cdn-avatars.huggingface.co/v1/production/uploads/1663421120944-noauth.png
Maria Wang
user
MariaW
65c43d0847a0ffffa9680449
Fedir Zubach
admin_assigned
2024-02-08T11:55:04.893Z
65364efd5f5e918bdc26c64c
/avatars/0b6c92c98e0dfc0ef437bb781dd51da5.svg
Fedir Zubach
user
fzubach
65c43d0847a0ffffa968044a
Hassan Mansoor
admin_assigned
2024-02-08T11:55:10.776Z
629098fd5463575364e7697a
/avatars/e45f3ee28bff3571f76caf847d3c36db.svg
Hassan Mansoor
user
HassanMansoor
65c43d0847a0ffffa968044b
Vincent Etter
admin_assigned
2024-02-08T11:55:16.541Z
6336dd00aeab734eec3f82f3
/avatars/460ef0ede333e4cacf97afd1f263e612.svg
Vincent Etter
user
w00ka1
65c43d0847a0ffffa968044c
Victor Cărbune
admin_assigned
2024-02-08T11:55:30.188Z
656054efc40c3a6a0d9d7b3e
/avatars/6d93e933df1cd59b836f5f79b5e938bf.svg
Victor Carbune
user
vcarbune
65c43d0847a0ffffa968044d
Jason Lin
claimed_verified
2024-02-09T08:05:54.738Z
635f700b398ff343c4fd5c3b
/avatars/d62ef6809b9d8da8713ecdafa36fb137.svg
Jason Lin
user
jaxball
65c43d0847a0ffffa968044e
Jindong Chen
admin_assigned
2024-02-08T11:55:52.413Z
635242b8b3678a43742baec6
/avatars/fc9f6d3922c76ab902cb11ed23554c54.svg
chenjindong
user
chenjindong
65c43d0847a0ffffa968044f
Abhanshu Sharma
ScreenAI: A Vision-Language Model for UI and Infographics Understanding
Screen user interfaces (UIs) and infographics, sharing similar visual
language and design principles, play important roles in human communication and
human-machine interaction. We introduce ScreenAI, a vision-language model that
specializes in UI and infographics understanding. Our model improves upon the
PaLI architecture with the flexible patching strategy of pix2struct and is
trained on a unique mixture of datasets. At the heart of this mixture is a
novel screen annotation task in which the model has to identify the type and
location of UI elements. We use these text annotations to describe screens to
Large Language Models and automatically generate question-answering (QA), UI
navigation, and summarization training datasets at scale. We run ablation
studies to demonstrate the impact of these design choices. At only 5B
parameters, ScreenAI achieves new state-of-the-artresults on UI- and
infographics-based tasks (Multi-page DocVQA, WebSRC, MoTIF and Widget
Captioning), and new best-in-class performance on others (Chart QA, DocVQA, and
InfographicVQA) compared to models of similar size. Finally, we release three
new datasets: one focused on the screen annotation task and two others focused
on question answering.
ScreenAI, a vision-language model optimized with a flexible patching strategy, achieves state-of-the-art performance on UI and infographic tasks with 5B parameters.
vision-language model PaLI architecture pix2struct flexible patching strategy screen annotation task UI elements question-answering UI navigation summarization Multi-page DocVQA WebSRC MoTIF Widget Captioning Chart QA DocVQA InfographicVQA