CogVLM: Visual Expert for Pretrained Language Models
We introduce CogVLM, a powerful open-source visual language foundation model.
Different from the popular shallow alignment method which maps image features
into the input space of language model, CogVLM bridges the gap between the
frozen pretrained language model and image encoder by a trainable visual expert
module in the attention and FFN layers. As a result, CogVLM enables deep fusion
of vision language features without sacrificing any performance on NLP tasks.
CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
6549c87ef6a1a3023567dd7c
Weihan Wang
claimed_verified
2025-07-02T12:23:58.023Z
63403c75b8b51e0098d3555c
/avatars/85725280b63788e387fe73319a54164d.svg
王维汉
user
mactavish91
6549c87ef6a1a3023567dd7d
Qingsong Lv
admin_assigned
2023-11-07T14:55:24.738Z
61bd8ac694bd578f7286181b
/avatars/ff14b64339217858927ae01c2b2fa28d.svg
Qingsong Lv
user
qingsonglv
6549c87ef6a1a3023567dd7e
Wenmeng Yu
claimed_verified
2025-07-02T09:01:42.430Z
649d3d271bafbcc83acec930
/avatars/0c42aabf4c6601686c22cc1308c318de.svg
Wenmeng Yu
user
iyuge2
6549c87ef6a1a3023567dd7f
Wenyi Hong
admin_assigned
2023-11-07T14:55:42.097Z
62ecd24cb8764c7738ef2793
/avatars/c1b80b5c55f9d652c1aaac7919e1fa32.svg
Wenyi Hong
user
wenyi
6549c87ef6a1a3023567dd80
Ji Qi
claimed_verified
2025-07-02T08:24:22.718Z
6864b9f90a269bace8c92164
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/zSeWHZRCZtzZ13CzFrr5u.png
Ji Qi
user
miracle11121
6549c87ef6a1a3023567dd81
Yan Wang
claimed_verified
2025-07-02T12:23:56.398Z
6441529fb3e14d9ba7b79d61
/avatars/79e81bbde65b5caaa6f7c000bba25296.svg
Yan
user
lykeven
6549c87ef6a1a3023567dd82
Junhui Ji
claimed_verified
2025-07-02T08:33:02.545Z
649cffffe89a765015a243a4
/avatars/20753e7ed68d1fad6ff633101b0ee2e4.svg
Junhui Ji
user
jasonnoy
6549c87ef6a1a3023567dd83
Zhuoyi Yang
admin_assigned
2024-08-29T09:10:37.036Z
6322e02496d6c8518eb05292
/avatars/6b5d317f9ccb0046e40918c7861b4cb5.svg
zhuoyi
user
keg-yzy
6549c87ef6a1a3023567dd84
Lei Zhao
6549c87ef6a1a3023567dd85
Xixuan Song
admin_assigned
2023-11-07T14:57:44.357Z
63f36b504745321de3510823
https://cdn-avatars.huggingface.co/v1/production/uploads/1679311624745-63f36b504745321de3510823.jpeg
Song XiXuan
user
DrSong
6549c87ef6a1a3023567dd86
Jiazheng Xu
admin_assigned
2023-11-07T14:58:10.155Z
62d7b131f6e8ba66107af761
/avatars/f1c5df47aef69c824fd166722df8f670.svg
Jiazheng Xu
user
xujz0703
6549c87ef6a1a3023567dd87
Bin Xu
6549c87ef6a1a3023567dd88
Juanzi Li
6549c87ef6a1a3023567dd89
Yuxiao Dong
admin_assigned
2023-11-07T14:58:50.620Z
640e73bdfdeaae1390857b62
/avatars/cd6779e30f716002a7838ed93d5c0754.svg
Yuxiao Dong
user
yuxiaod
6549c87ef6a1a3023567dd8a
Ming Ding
6549c87ef6a1a3023567dd8b
Jie Tang
admin_assigned
2023-11-07T14:59:35.474Z
640dff05474aa6f89556677e
/avatars/1b4591c7322d649c797b3125148f1915.svg
Jie Tang
user
jerytang
CogVLM: Visual Expert for Pretrained Language Models
We introduce CogVLM, a powerful open-source visual language foundation model.
Different from the popular shallow alignment method which maps image features
into the input space of language model, CogVLM bridges the gap between the
frozen pretrained language model and image encoder by a trainable visual expert
module in the attention and FFN layers. As a result, CogVLM enables deep fusion
of vision language features without sacrificing any performance on NLP tasks.
CogVLM-17B achieves state-of-the-art performance on 10 classic cross-modal
benchmarks, including NoCaps, Flicker30k captioning, RefCOCO, RefCOCO+,
RefCOCOg, Visual7W, GQA, ScienceQA, VizWiz VQA and TDIUC, and ranks the 2nd on
VQAv2, OKVQA, TextVQA, COCO captioning, etc., surpassing or matching PaLI-X
55B. Codes and checkpoints are available at https://github.com/THUDM/CogVLM.
CogVLM, a visual language foundation model, uses a trainable visual expert module to deeply integrate vision and language without compromising NLP performance.
visual language foundation model shallow alignment method trainable visual expert module attention layers FFN layers deep fusion cross-modal benchmarks NoCaps Flicker30k captioning RefCOCO RefCOCO+ RefCOCOg Visual7W GQA ScienceQA VizWiz VQA TDIUC VQAv2 OKVQA TextVQA COCO captioning PaLI-X