title: ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
summary: Perception and understanding are two pillars of computer vision. While
  multimodal large language models (MLLM) have demonstrated remarkable visual
  understanding capabilities, they arguably lack accurate perception abilities,
  e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
  the COCO dataset, limiting many tasks requiring the combination of perception
  and understanding. In this work, we aim to bridge this perception gap from both
  model designing and data development perspectives. We first introduce ChatRex,
  an MLLM with a decoupled perception design. Instead of having the LLM directly
  predict box coordinates, we feed the output boxes from a universal proposal
  network into the LLM, allowing it to output the corresponding box indices to
  represent its detection results, turning the regression task into a
  retrieval-based task that LLM handles more proficiently. From the data
  perspective, we build a fully automated data engine and construct the
  Rexverse-2M dataset which possesses multiple granularities to support the joint
  training of perception and understanding. After standard two-stage training,
  ChatRex demonstrates strong perception capabilities while preserving multimodal
  understanding performance. The combination of these two capabilities
  simultaneously unlocks many attractive applications, demonstrating the
  complementary roles of both perception and understanding in MLLM. Code is
  available at https://github.com/IDEA-Research/ChatRex.
paper_authors._id: 674805864fedd3964253eadf
paper_authors.name: Qing Jiang
paper_authors._id: 674805864fedd3964253eae0
paper_authors.name: Gen luo
paper_authors._id: 674805864fedd3964253eae1
paper_authors.name: Yuqin Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T13:12:28.350Z
paper_authors._id: 64672f4aa9b4610868a0407e
paper_authors.avatarUrl: /avatars/2e964e7edd2d1ebaa2b8597992090925.svg
paper_authors.fullname: Yuqin Yang
paper_authors.type: user
paper_authors.user: CRIS-Yang
paper_authors._id: 674805864fedd3964253eae2
paper_authors.name: Yuda Xiong
paper_authors._id: 674805864fedd3964253eae3
paper_authors.name: Yihao Chen
paper_authors._id: 674805864fedd3964253eae4
paper_authors.name: Zhaoyang Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-28T12:29:31.995Z
paper_authors._id: 669f0abdc13da2bbe268c5c3
paper_authors.avatarUrl: /avatars/87961bfd01c53c420ae13133d643f22c.svg
paper_authors.fullname: Zhaoyang Zeng
paper_authors.type: user
paper_authors.user: zengzhaoyang
paper_authors._id: 674805864fedd3964253eae5
paper_authors.name: Tianhe Ren
paper_authors._id: 674805864fedd3964253eae6
paper_authors.name: Lei Zhang
paper_title: ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
paper_summary: Perception and understanding are two pillars of computer vision. While
  multimodal large language models (MLLM) have demonstrated remarkable visual
  understanding capabilities, they arguably lack accurate perception abilities,
  e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
  the COCO dataset, limiting many tasks requiring the combination of perception
  and understanding. In this work, we aim to bridge this perception gap from both
  model designing and data development perspectives. We first introduce ChatRex,
  an MLLM with a decoupled perception design. Instead of having the LLM directly
  predict box coordinates, we feed the output boxes from a universal proposal
  network into the LLM, allowing it to output the corresponding box indices to
  represent its detection results, turning the regression task into a
  retrieval-based task that LLM handles more proficiently. From the data
  perspective, we build a fully automated data engine and construct the
  Rexverse-2M dataset which possesses multiple granularities to support the joint
  training of perception and understanding. After standard two-stage training,
  ChatRex demonstrates strong perception capabilities while preserving multimodal
  understanding performance. The combination of these two capabilities
  simultaneously unlocks many attractive applications, demonstrating the
  complementary roles of both perception and understanding in MLLM. Code is
  available at https://github.com/IDEA-Research/ChatRex.
paper_ai_keywords: ChatRex multimodal large language models (MLLM) COCO dataset decoupled perception design universal proposal network box coordinates box indices retrieval-based task standard two-stage training Rexverse-2M dataset perception capabilities multimodal understanding performance