title: Exploring Multi-Grained Concept Annotations for Multimodal Large
    Language Models
summary: Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
  pre-training solely on coarse-grained concept annotations (e.g., image
  captions). We hypothesize that integrating fine-grained concept annotations
  (e.g., object labels and object regions) will further improve performance, as
  both data granularities complement each other in terms of breadth and depth in
  concept representation. We introduce a new dataset featuring Multimodal
  Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
  explore the impact of different data recipes on multimodal comprehension and
  generation. Our analyses reveal that multi-grained concept annotations
  integrate and complement each other, under our structured template and a
  general MLLM framework. We clearly explore and demonstrate the potential of
  MMGiC to help MLLMs better locate and learn concepts, aligning vision and
  language at multiple granularities. We further validate our hypothesis by
  investigating the fair comparison and effective collaboration between MMGiC and
  image--caption data on 12 multimodal comprehension and generation benchmarks,
  e.g., their appropriate combination achieve 3.95% and 2.34% absolute
  improvements over image--caption data alone on POPE and SEED-Bench. Code, data
  and models will be available at https://github.com/LooperXX/MMGiC.
paper_authors._id: 6757a3cef4cf69dc8e379cae
paper_authors.name: Xiao Xu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-10T10:25:23.468Z
paper_authors._id: 616fb788e2ad27af26561b1a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1675485317568-616fb788e2ad27af26561b1a.jpeg
paper_authors.fullname: Xiao Xu
paper_authors.type: user
paper_authors.user: LooperXX
paper_authors._id: 6757a3cef4cf69dc8e379caf
paper_authors.name: Tianhao Niu
paper_authors._id: 6757a3cef4cf69dc8e379cb0
paper_authors.name: Yuxi Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:11.040Z
paper_authors._id: 647730d403fe88eff54d9b24
paper_authors.avatarUrl: /avatars/66a41a42a4f9adaca7477b73ab1a0c0e.svg
paper_authors.fullname: Sigrid Xie
paper_authors.type: user
paper_authors.user: yuxixie
paper_authors._id: 6757a3cef4cf69dc8e379cb1
paper_authors.name: Libo Qin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:17.307Z
paper_authors._id: 63e8f4c0ccae1fe5c61b636f
paper_authors.avatarUrl: /avatars/013e124e37b6194d5564a1db9aad2b20.svg
paper_authors.fullname: qin
paper_authors.type: user
paper_authors.user: liboqin
paper_authors._id: 6757a3cef4cf69dc8e379cb2
paper_authors.name: Wanxiang Che
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:23.487Z
paper_authors._id: 6325418ae5d49a0d40f66176
paper_authors.avatarUrl: /avatars/59a0e29ce09e9113edc53cebb728b099.svg
paper_authors.fullname: Wanxiang Che
paper_authors.type: user
paper_authors.user: wanxiangche
paper_authors._id: 6757a3cef4cf69dc8e379cb3
paper_authors.name: Min-Yen Kan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-10T16:16:28.903Z
paper_authors._id: 628f137053028c04a3b750e4
paper_authors.avatarUrl: /avatars/85d174cc041ab91e724bb9d9d4e46c88.svg
paper_authors.fullname: Min-Yen Kan
paper_authors.type: user
paper_authors.user: knmnyn
paper_title: Exploring Multi-Grained Concept Annotations for Multimodal Large
    Language Models
paper_summary: Multimodal Large Language Models (MLLMs) excel in vision--language tasks by
  pre-training solely on coarse-grained concept annotations (e.g., image
  captions). We hypothesize that integrating fine-grained concept annotations
  (e.g., object labels and object regions) will further improve performance, as
  both data granularities complement each other in terms of breadth and depth in
  concept representation. We introduce a new dataset featuring Multimodal
  Multi-Grained Concept annotations (MMGiC) for MLLMs. In constructing MMGiC, we
  explore the impact of different data recipes on multimodal comprehension and
  generation. Our analyses reveal that multi-grained concept annotations
  integrate and complement each other, under our structured template and a
  general MLLM framework. We clearly explore and demonstrate the potential of
  MMGiC to help MLLMs better locate and learn concepts, aligning vision and
  language at multiple granularities. We further validate our hypothesis by
  investigating the fair comparison and effective collaboration between MMGiC and
  image--caption data on 12 multimodal comprehension and generation benchmarks,
  e.g., their appropriate combination achieve 3.95% and 2.34% absolute
  improvements over image--caption data alone on POPE and SEED-Bench. Code, data
  and models will be available at https://github.com/LooperXX/MMGiC.
paper_ai_keywords: Multimodal Large Language Models (MLLMs) vision--Language Tasks pre-training coarse-grained concept annotations fine-grained concept annotations object labels object regions Multimodal Multi-Grained Concept trainings (MMGiC) multimodal comprehension multimodal generation structured template concept representation MLLM framework visual concepts language concepts multimodal benchmarks POPE SEED-Bench