title: CogVLM: Visual Expert for Pretrained Language Models
summary: We introduce CogVLM, a powerful open-source visual language foundation model.
  Different from the popular shallow alignment method which maps image features
  into the input space of language model, CogVLM bridges the gap between the
  frozen pretrained language model and image encoder by a trainable visual expert
  module in the attention and FFN layers. As a result, CogVLM enables deep fusion
  of vision language features without sacrificing any performance on NLP tasks.
  CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
  benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
  RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
  VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
  55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
paper_authors._id: 6549c87ef6a1a3023567dd7c
paper_authors.name: Weihan Wang
paper_authors._id: 6549c87ef6a1a3023567dd7d
paper_authors.name: Qingsong Lv
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:55:24.738Z
paper_authors._id: 61bd8ac694bd578f7286181b
paper_authors.avatarUrl: /avatars/ff14b64339217858927ae01c2b2fa28d.svg
paper_authors.fullname: Qingsong Lv
paper_authors.type: user
paper_authors.user: qingsonglv
paper_authors._id: 6549c87ef6a1a3023567dd7e
paper_authors.name: Wenmeng Yu
paper_authors._id: 6549c87ef6a1a3023567dd7f
paper_authors.name: Wenyi Hong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:55:42.097Z
paper_authors._id: 62ecd24cb8764c7738ef2793
paper_authors.avatarUrl: /avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
paper_authors.fullname: Wenyi Hong
paper_authors.type: user
paper_authors.user: wenyi
paper_authors._id: 6549c87ef6a1a3023567dd80
paper_authors.name: Ji Qi
paper_authors._id: 6549c87ef6a1a3023567dd81
paper_authors.name: Yan Wang
paper_authors._id: 6549c87ef6a1a3023567dd82
paper_authors.name: Junhui Ji
paper_authors._id: 6549c87ef6a1a3023567dd83
paper_authors.name: Zhuoyi Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-29T09:10:37.036Z
paper_authors._id: 6322e02496d6c8518eb05292
paper_authors.avatarUrl: /avatars/6b5d317f9ccb0046e40918c7861b4cb5.svg
paper_authors.fullname: zhuoyi
paper_authors.type: user
paper_authors.user: keg-yzy
paper_authors._id: 6549c87ef6a1a3023567dd84
paper_authors.name: Lei Zhao
paper_authors._id: 6549c87ef6a1a3023567dd85
paper_authors.name: Xixuan Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:57:44.357Z
paper_authors._id: 63f36b504745321de3510823
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679311624745-63f36b504745321de3510823.jpeg
paper_authors.fullname: Song XiXuan
paper_authors.type: user
paper_authors.user: DrSong
paper_authors._id: 6549c87ef6a1a3023567dd86
paper_authors.name: Jiazheng Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:58:10.155Z
paper_authors._id: 62d7b131f6e8ba66107af761
paper_authors.avatarUrl: /avatars/f1c5df47aef69c824fd166722df8f670.svg
paper_authors.fullname: Jiazheng Xu
paper_authors.type: user
paper_authors.user: xujz0703
paper_authors._id: 6549c87ef6a1a3023567dd87
paper_authors.name: Bin Xu
paper_authors._id: 6549c87ef6a1a3023567dd88
paper_authors.name: Juanzi Li
paper_authors._id: 6549c87ef6a1a3023567dd89
paper_authors.name: Yuxiao Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:58:50.620Z
paper_authors._id: 640e73bdfdeaae1390857b62
paper_authors.avatarUrl: /avatars/cd6779e30f716002a7838ed93d5c0754.svg
paper_authors.fullname: Yuxiao Dong
paper_authors.type: user
paper_authors.user: yuxiaod
paper_authors._id: 6549c87ef6a1a3023567dd8a
paper_authors.name: Ming Ding
paper_authors._id: 6549c87ef6a1a3023567dd8b
paper_authors.name: Jie Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-07T14:59:35.474Z
paper_authors._id: 640dff05474aa6f89556677e
paper_authors.avatarUrl: /avatars/1b4591c7322d649c797b3125148f1915.svg
paper_authors.fullname: Jie Tang
paper_authors.type: user
paper_authors.user: jerytang
paper_title: CogVLM: Visual Expert for Pretrained Language Models
paper_summary: We introduce CogVLM, a powerful open-source visual language foundation model.
  Different from the popular shallow alignment method which maps image features
  into the input space of language model, CogVLM bridges the gap between the
  frozen pretrained language model and image encoder by a trainable visual expert
  module in the attention and FFN layers. As a result, CogVLM enables deep fusion
  of vision language features without sacrificing any performance on NLP tasks.
  CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
  benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
  RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
  VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
  55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
paper_ai_keywords: visual language foundation model shallow alignment method image features input space pretrained language model image encoder trainable visual expert module attention FFN layers deep fusion cross-modal benchmarks NoCaps Flicker30k captioning RefCOCO RefCOCO+ RefCOCOg Visual7W GQA ScienceQA VizWiz VQA TDIUC VQAv2 OKVQA TextVQA COCO captioning PaLI-X