title: OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and
    Understanding
summary: Current universal segmentation methods demonstrate strong capabilities in
  pixel-level image and video understanding. However, they lack reasoning
  abilities and cannot be controlled via text instructions. In contrast, large
  vision-language multimodal models exhibit powerful vision-based conversation
  and reasoning capabilities but lack pixel-level understanding and have
  difficulty accepting visual prompts for flexible user interaction. This paper
  proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
  vision understanding with reasoning abilities. It can accept various visual and
  text prompts for flexible user interaction. Specifically, we use a universal
  segmentation method as the visual encoder, integrating image information,
  perception priors, and visual prompts into visual tokens provided to the LLM.
  The LLM is responsible for understanding the user's text instructions and
  providing text responses and pixel-level segmentation results based on the
  visual information. We propose perception prior embedding to better integrate
  perception priors with image features. OMG-LLaVA achieves image-level,
  object-level, and pixel-level reasoning and understanding in a single model,
  matching or surpassing the performance of specialized methods on multiple
  benchmarks. Rather than using LLM to connect each specialist, our work aims at
  end-to-end training on one encoder, one decoder, and one LLM. The code and
  model have been released for further research.
paper_authors._id: 667e26bdb0f62e5e076bc54f
paper_authors.name: Tao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-10T09:21:03.597Z
paper_authors._id: 6436435c2d0ed796669258d3
paper_authors.avatarUrl: /avatars/d357378eb039391e8ce74bbd84b80d07.svg
paper_authors.fullname: zhangtao
paper_authors.type: user
paper_authors.user: zhangtao-whu
paper_authors._id: 667e26bdb0f62e5e076bc550
paper_authors.name: Xiangtai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:20:50.107Z
paper_authors._id: 63958b4414513eaf9029ebf1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/U1g5H071pWRswGAG9UTpo.png
paper_authors.fullname: Xiangtai Li
paper_authors.type: user
paper_authors.user: LXT
paper_authors._id: 667e26bdb0f62e5e076bc551
paper_authors.name: Hao Fei
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-15T09:46:32.950Z
paper_authors._id: 647773a1168cb428e00e9a8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
paper_authors.fullname: Hao Fei
paper_authors.type: user
paper_authors.user: scofield7419
paper_authors._id: 667e26bdb0f62e5e076bc552
paper_authors.name: Haobo Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:21:29.050Z
paper_authors._id: 6391e41f2e73987364e6bcb2
paper_authors.avatarUrl: /avatars/d09a9ee329bb8c3a9e2929d67d24e97d.svg
paper_authors.fullname: Haobo Yuan
paper_authors.type: user
paper_authors.user: HarborYuan
paper_authors._id: 667e26bdb0f62e5e076bc553
paper_authors.name: Shengqiong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-28T09:25:21.171Z
paper_authors._id: 64c139d867eff857ea51caa8
paper_authors.avatarUrl: /avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
paper_authors.fullname: Shengqiong Wu
paper_authors.type: user
paper_authors.user: ChocoWu
paper_authors._id: 667e26bdb0f62e5e076bc554
paper_authors.name: Shunping Ji
paper_authors._id: 667e26bdb0f62e5e076bc555
paper_authors.name: Chen Change Loy
paper_authors._id: 667e26bdb0f62e5e076bc556
paper_authors.name: Shuicheng Yan
paper_title: OMG-LLaVA: Bridging Image-level, Object-level, Pixel-level Reasoning and
    Understanding
paper_summary: Current universal segmentation methods demonstrate strong capabilities in
  pixel-level image and video understanding. However, they lack reasoning
  abilities and cannot be controlled via text instructions. In contrast, large
  vision-language multimodal models exhibit powerful vision-based conversation
  and reasoning capabilities but lack pixel-level understanding and have
  difficulty accepting visual prompts for flexible user interaction. This paper
  proposes OMG-LLaVA, a new and elegant framework combining powerful pixel-level
  vision understanding with reasoning abilities. It can accept various visual and
  text prompts for flexible user interaction. Specifically, we use a universal
  segmentation method as the visual encoder, integrating image information,
  perception priors, and visual prompts into visual tokens provided to the LLM.
  The LLM is responsible for understanding the user's text instructions and
  providing text responses and pixel-level segmentation results based on the
  visual information. We propose perception prior embedding to better integrate
  perception priors with image features. OMG-LLaVA achieves image-level,
  object-level, and pixel-level reasoning and understanding in a single model,
  matching or surpassing the performance of specialized methods on multiple
  benchmarks. Rather than using LLM to connect each specialist, our work aims at
  end-to-end training on one encoder, one decoder, and one LLM. The code and
  model have been released for further research.
paper_ai_keywords: segmentation method vision-language multimodal models vision-based conversation reasoning abilities visual prompts universal segmentation method visual encoder perception priors visual tokens LLM (Large Language Model) perception prior embedding image-level reasoning object-level reasoning pixel-level reasoning end-to-end training