Exploring Multi-Grained Concept Annotations for Multimodal Large
  Language Models
Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
pre-training solely on coarse-grained concept annotations (e.g., image
captions). We hypothesize that integrating fine-grained concept annotations
(e.g., object labels and object regions) will further improve performance, as
both data granularities complement each other in terms of breadth and depth in
concept representation. We introduce a new dataset featuring Multimodal
Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
explore the impact of different data recipes on multimodal comprehension and
generation. Our analyses reveal that multi-grained concept annotations
integrate and complement each other, under our structured template and a
general MLLM framework. We clearly explore and demonstrate the potential of
MMGiC to help MLLMs better locate and learn concepts, aligning vision and
language at multiple granularities. We further validate our hypothesis by
investigating the fair comparison and effective collaboration between MMGiC and
image--caption data on 12 multimodal comprehension and generation benchmarks,
e.g., their appropriate combination achieve 3.95% and 2.34% absolute
improvements over image--caption data alone on POPE and SEED-Bench. Code, data
and models will be available at https://github.com/LooperXX/MMGiC.
6757a3cef4cf69dc8e379cae
Xiao Xu
claimed_verified
2024-12-10T10:25:23.468Z
616fb788e2ad27af26561b1a
https://cdn-avatars.huggingface.co/v1/production/uploads/1675485317568-616fb788e2ad27af26561b1a.jpeg
Xiao Xu
user
LooperXX
6757a3cef4cf69dc8e379caf
Tianhao Niu
6757a3cef4cf69dc8e379cb0
Yuxi Xie
admin_assigned
2024-12-10T16:16:11.040Z
647730d403fe88eff54d9b24
/avatars/66a41a42a4f9adaca7477b73ab1a0c0e.svg
Sigrid Xie
user
yuxixie
6757a3cef4cf69dc8e379cb1
Libo Qin
admin_assigned
2024-12-10T16:16:17.307Z
63e8f4c0ccae1fe5c61b636f
/avatars/013e124e37b6194d5564a1db9aad2b20.svg
qin
user
liboqin
6757a3cef4cf69dc8e379cb2
Wanxiang Che
admin_assigned
2024-12-10T16:16:23.487Z
6325418ae5d49a0d40f66176
/avatars/59a0e29ce09e9113edc53cebb728b099.svg
Wanxiang Che
user
wanxiangche
6757a3cef4cf69dc8e379cb3
Min-Yen Kan
admin_assigned
2024-12-10T16:16:28.903Z
628f137053028c04a3b750e4
/avatars/85d174cc041ab91e724bb9d9d4e46c88.svg
Min-Yen Kan
user
knmnyn
Exploring Multi-Grained Concept Annotations for Multimodal Large
  Language Models
Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
pre-training solely on coarse-grained concept annotations (e.g., image
captions). We hypothesize that integrating fine-grained concept annotations
(e.g., object labels and object regions) will further improve performance, as
both data granularities complement each other in terms of breadth and depth in
concept representation. We introduce a new dataset featuring Multimodal
Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
explore the impact of different data recipes on multimodal comprehension and
generation. Our analyses reveal that multi-grained concept annotations
integrate and complement each other, under our structured template and a
general MLLM framework. We clearly explore and demonstrate the potential of
MMGiC to help MLLMs better locate and learn concepts, aligning vision and
language at multiple granularities. We further validate our hypothesis by
investigating the fair comparison and effective collaboration between MMGiC and
image--caption data on 12 multimodal comprehension and generation benchmarks,
e.g., their appropriate combination achieve 3.95% and 2.34% absolute
improvements over image--caption data alone on POPE and SEED-Bench. Code, data
and models will be available at https://github.com/LooperXX/MMGiC.
Integrating fine-grained concept annotations enhances multimodal large language models, improving their performance in vision-language tasks by aligning vision and language at multiple granularities.
Multimodal Large Language Models MLLMs fine-grained concept annotations Multimodal Multi-Grained Concept annotations MMGiC multimodal comprehension multimodal generation POPE SEED-Bench