title: LEGO:Language Enhanced Multi-modal Grounding Model
summary: Multi-modal large language models have demonstrated impressive performance
  across various tasks in different modalities. However, existing multi-modal
  models primarily emphasize capturing global information within each modality
  while neglecting the importance of perceiving local information across
  modalities. Consequently, these models lack the ability to effectively
  understand the fine-grained details of input data, limiting their performance
  in tasks that require a more nuanced understanding. To address this limitation,
  there is a compelling need to develop models that enable fine-grained
  understanding across multiple modalities, thereby enhancing their applicability
  to a wide range of tasks. In this paper, we propose LEGO, a language enhanced
  multi-modal grounding model. Beyond capturing global information like other
  multi-modal models, our proposed model excels at tasks demanding a detailed
  understanding of local information within the input. It demonstrates precise
  identification and localization of specific regions in images or moments in
  videos. To achieve this objective, we design a diversified dataset construction
  pipeline, resulting in a multi-modal, multi-granularity dataset for model
  training. The code, dataset, and demo of our model can be found at https:
  //github.com/lzw-lzw/LEGO.
paper_authors._id: 65a0a7315dfd8b9b1f288fe6
paper_authors.name: Zhaowei Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-30T08:10:45.647Z
paper_authors._id: 6469eff38eaf6dcec22e0a8e
paper_authors.avatarUrl: /avatars/0e73f4fba981fa0a93b6ae44ace9dbd7.svg
paper_authors.fullname: lizhaowei
paper_authors.type: user
paper_authors.user: zwli
paper_authors._id: 65a0a7315dfd8b9b1f288fe7
paper_authors.name: Qi Xu
paper_authors._id: 65a0a7315dfd8b9b1f288fe8
paper_authors.name: Dong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:14:18.120Z
paper_authors._id: 6377596f82ce7c3b8a528f72
paper_authors.avatarUrl: /avatars/f55676883e054d7e4a3b96ba1e752a80.svg
paper_authors.fullname: Dong Zhang
paper_authors.type: user
paper_authors.user: nutation
paper_authors._id: 65a0a7315dfd8b9b1f288fe9
paper_authors.name: Hang Song
paper_authors._id: 65a0a7315dfd8b9b1f288fea
paper_authors.name: Yiqing Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:30:47.851Z
paper_authors._id: 63034e3e0ddba5a5b9f47814
paper_authors.avatarUrl: /avatars/c740335400421904c06d1c576cb8ae87.svg
paper_authors.fullname: Yiqing Cai
paper_authors.type: user
paper_authors.user: SmallCai
paper_authors._id: 65a0a7315dfd8b9b1f288feb
paper_authors.name: Qi Qi
paper_authors._id: 65a0a7315dfd8b9b1f288fec
paper_authors.name: Ran Zhou
paper_authors._id: 65a0a7315dfd8b9b1f288fed
paper_authors.name: Junting Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-12T10:30:22.022Z
paper_authors._id: 639c2d36e62c431aa8d834a7
paper_authors.avatarUrl: /avatars/a23d57dcb190711494a2a7e75e4cb9a8.svg
paper_authors.fullname: Junting Pan
paper_authors.type: user
paper_authors.user: juntingpan
paper_authors._id: 65a0a7315dfd8b9b1f288fee
paper_authors.name: Zefeng Li
paper_authors._id: 65a0a7315dfd8b9b1f288fef
paper_authors.name: Van Tu Vu
paper_authors._id: 65a0a7315dfd8b9b1f288ff0
paper_authors.name: Zhida Huang
paper_authors._id: 65a0a7315dfd8b9b1f288ff1
paper_authors.name: Tao Wang
paper_title: LEGO:Language Enhanced Multi-modal Grounding Model
paper_summary: Multi-modal large language models have demonstrated impressive performance
  across various tasks in different modalities. However, existing multi-modal
  models primarily emphasize capturing global information within each modality
  while neglecting the importance of perceiving local information across
  modalities. Consequently, these models lack the ability to effectively
  understand the fine-grained details of input data, limiting their performance
  in tasks that require a more nuanced understanding. To address this limitation,
  there is a compelling need to develop models that enable fine-grained
  understanding across multiple modalities, thereby enhancing their applicability
  to a wide range of tasks. In this paper, we propose LEGO, a language enhanced
  multi-modal grounding model. Beyond capturing global information like other
  multi-modal models, our proposed model excels at tasks demanding a detailed
  understanding of local information within the input. It demonstrates precise
  identification and localization of specific regions in images or moments in
  videos. To achieve this objective, we design a diversified dataset construction
  pipeline, resulting in a multi-modal, multi-granularity dataset for model
  training. The code, dataset, and demo of our model can be found at https:
  //github.com/lzw-lzw/LEGO.
paper_ai_keywords: language enhanced multi-modal grounding model fine-grained understanding precise identification localization multi-modal, multi-granularity dataset