CogAgent: A Visual Language Model for GUI Agents
People are spending an enormous amount of time on digital devices through
graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
language models (LLMs) such as ChatGPT can assist people in tasks like writing
emails, but struggle to understand and interact with GUIs, thus limiting their
potential to increase automation levels. In this paper, we introduce CogAgent,
an 18-billion-parameter visual language model (VLM) specializing in GUI
understanding and navigation. By utilizing both low-resolution and
high-resolution image encoders, CogAgent supports input at a resolution of
1120*1120, enabling it to recognize tiny page elements and text. As a
generalist visual language model, CogAgent achieves the state of the art on
five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
only screenshots as input, outperforms LLM-based methods that consume extracted
HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
advancing the state of the art. The model and codes are available at
https://github.com/THUDM/CogVLM.
657bcbace34a7de14b0c769e
Wenyi Hong
admin_assigned
2023-12-15T10:31:41.388Z
62ecd24cb8764c7738ef2793
/avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
Wenyi Hong
user
wenyi
657bcbace34a7de14b0c769f
Weihan Wang
claimed_verified
2025-07-02T12:23:54.077Z
63403c75b8b51e0098d3555c
/avatars/85725280b63788e387fe73319a54164d.svg
王维汉
user
mactavish91
657bcbace34a7de14b0c76a0
Qingsong Lv
admin_assigned
2023-12-15T10:32:22.947Z
61bd8ac694bd578f7286181b
/avatars/ff14b64339217858927ae01c2b2fa28d.svg
Qingsong Lv
user
qingsonglv
657bcbace34a7de14b0c76a1
Jiazheng Xu
admin_assigned
2023-12-15T10:32:44.638Z
62d7b131f6e8ba66107af761
/avatars/f1c5df47aef69c824fd166722df8f670.svg
Jiazheng Xu
user
xujz0703
657bcbace34a7de14b0c76a2
Wenmeng Yu
admin_assigned
2023-12-15T10:32:57.012Z
655bcec1a6a120a5c2ecd997
/avatars/6f4ba5013a45ce4ad65e873bd644c8fe.svg
Meng
user
YuwenMeng
657bcbace34a7de14b0c76a3
Junhui Ji
claimed_verified
2025-07-02T08:33:01.111Z
649cffffe89a765015a243a4
/avatars/20753e7ed68d1fad6ff633101b0ee2e4.svg
Junhui Ji
user
jasonnoy
657bcbace34a7de14b0c76a4
Yan Wang
claimed_verified
2025-07-02T12:23:51.668Z
6441529fb3e14d9ba7b79d61
/avatars/79e81bbde65b5caaa6f7c000bba25296.svg
Yan
user
lykeven
657bcbace34a7de14b0c76a5
Zihan Wang
657bcbace34a7de14b0c76a6
Yuxiao Dong
admin_assigned
2023-12-15T10:36:18.257Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
657bcbace34a7de14b0c76a7
Ming Ding
657bcbace34a7de14b0c76a8
Jie Tang
admin_assigned
2023-12-15T10:43:45.989Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
CogAgent: A Visual Language Model for GUI Agents
People are spending an enormous amount of time on digital devices through
graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
language models (LLMs) such as ChatGPT can assist people in tasks like writing
emails, but struggle to understand and interact with GUIs, thus limiting their
potential to increase automation levels. In this paper, we introduce CogAgent,
an 18-billion-parameter visual language model (VLM) specializing in GUI
understanding and navigation. By utilizing both low-resolution and
high-resolution image encoders, CogAgent supports input at a resolution of
1120*1120, enabling it to recognize tiny page elements and text. As a
generalist visual language model, CogAgent achieves the state of the art on
five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
only screenshots as input, outperforms LLM-based methods that consume extracted
HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
advancing the state of the art. The model and codes are available at
https://github.com/THUDM/CogVLM.
CogAgent, a visual language model with strong GUI understanding and navigation capabilities, outperforms LLM-based methods in both PC and Android GUI tasks using only screenshots.
visual language model GUI understanding navigation VLM image encoders VQAv2 OK-VQA Text-VQA ST-VQA ChartQA infoVQA DocVQA MM-Vet POPE