title: CogAgent: A Visual Language Model for GUI Agents
summary: People are spending an enormous amount of time on digital devices through
  graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
  language models (LLMs) such as ChatGPT can assist people in tasks like writing
  emails, but struggle to understand and interact with GUIs, thus limiting their
  potential to increase automation levels. In this paper, we introduce CogAgent,
  an 18-billion-parameter visual language model (VLM) specializing in GUI
  understanding and navigation. By utilizing both low-resolution and
  high-resolution image encoders, CogAgent supports input at a resolution of
  1120*1120, enabling it to recognize tiny page elements and text. As a
  generalist visual language model, CogAgent achieves the state of the art on
  five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
  Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
  only screenshots as input, outperforms LLM-based methods that consume extracted
  HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
  advancing the state of the art. The model and codes are available at
  https://github.com/THUDM/CogVLM.
paper_authors._id: 657bcbace34a7de14b0c769e
paper_authors.name: Wenyi Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:31:41.388Z
paper_authors._id: 62ecd24cb8764c7738ef2793
paper_authors.avatarUrl: /avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
paper_authors.fullname: Wenyi Hong
paper_authors.type: user
paper_authors.user: wenyi
paper_authors._id: 657bcbace34a7de14b0c769f
paper_authors.name: Weihan Wang
paper_authors._id: 657bcbace34a7de14b0c76a0
paper_authors.name: Qingsong Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:32:22.947Z
paper_authors._id: 61bd8ac694bd578f7286181b
paper_authors.avatarUrl: /avatars/ff14b64339217858927ae01c2b2fa28d.svg
paper_authors.fullname: Qingsong Lv
paper_authors.type: user
paper_authors.user: qingsonglv
paper_authors._id: 657bcbace34a7de14b0c76a1
paper_authors.name: Jiazheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:32:44.638Z
paper_authors._id: 62d7b131f6e8ba66107af761
paper_authors.avatarUrl: /avatars/f1c5df47aef69c824fd166722df8f670.svg
paper_authors.fullname: Jiazheng Xu
paper_authors.type: user
paper_authors.user: xujz0703
paper_authors._id: 657bcbace34a7de14b0c76a2
paper_authors.name: Wenmeng Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:32:57.012Z
paper_authors._id: 655bcec1a6a120a5c2ecd997
paper_authors.avatarUrl: /avatars/6f4ba5013a45ce4ad65e873bd644c8fe.svg
paper_authors.fullname: Meng
paper_authors.type: user
paper_authors.user: YuwenMeng
paper_authors._id: 657bcbace34a7de14b0c76a3
paper_authors.name: Junhui Ji
paper_authors._id: 657bcbace34a7de14b0c76a4
paper_authors.name: Yan Wang
paper_authors._id: 657bcbace34a7de14b0c76a5
paper_authors.name: Zihan Wang
paper_authors._id: 657bcbace34a7de14b0c76a6
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:36:18.257Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 657bcbace34a7de14b0c76a7
paper_authors.name: Ming Ding
paper_authors._id: 657bcbace34a7de14b0c76a8
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:43:45.989Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: CogAgent: A Visual Language Model for GUI Agents
paper_summary: People are spending an enormous amount of time on digital devices through
  graphical user interfaces (GUIs), e.g., computer or smartphone screens. Large
  language models (LLMs) such as ChatGPT can assist people in tasks like writing
  emails, but struggle to understand and interact with GUIs, thus limiting their
  potential to increase automation levels. In this paper, we introduce CogAgent,
  an 18-billion-parameter visual language model (VLM) specializing in GUI
  understanding and navigation. By utilizing both low-resolution and
  high-resolution image encoders, CogAgent supports input at a resolution of
  1120*1120, enabling it to recognize tiny page elements and text. As a
  generalist visual language model, CogAgent achieves the state of the art on
  five text-rich and four general VQA benchmarks, including VQAv2, OK-VQA,
  Text-VQA, ST-VQA, ChartQA, infoVQA, DocVQA, MM-Vet, and POPE. CogAgent, using
  only screenshots as input, outperforms LLM-based methods that consume extracted
  HTML text on both PC and Android GUI navigation tasks -- Mind2Web and AITW,
  advancing the state of the art. The model and codes are available at
  https://github.com/THUDM/CogVLM.
paper_ai_keywords: visual language model (VLM) GUI understanding GUI navigation image encoders VQA benchmarks VQAv2 OK-VQA Text-VQA ST-VQA ChartQA infoVQA DocVQA MM-Vet POPE screenshots