ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Perception and understanding are two pillars of computer vision. While
multimodal large language models (MLLM) have demonstrated remarkable visual
understanding capabilities, they arguably lack accurate perception abilities,
e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
the COCO dataset, limiting many tasks requiring the combination of perception
and understanding. In this work, we aim to bridge this perception gap from both
model designing and data development perspectives. We first introduce ChatRex,
an MLLM with a decoupled perception design. Instead of having the LLM directly
predict box coordinates, we feed the output boxes from a universal proposal
network into the LLM, allowing it to output the corresponding box indices to
represent its detection results, turning the regression task into a
retrieval-based task that LLM handles more proficiently. From the data
perspective, we build a fully automated data engine and construct the
Rexverse-2M dataset which possesses multiple granularities to support the joint
training of perception and understanding. After standard two-stage training,
ChatRex demonstrates strong perception capabilities while preserving multimodal
understanding performance. The combination of these two capabilities
simultaneously unlocks many attractive applications, demonstrating the
complementary roles of both perception and understanding in MLLM. Code is
available at https://github.com/IDEA-Research/ChatRex.
674805864fedd3964253eadf
Qing Jiang
674805864fedd3964253eae0
Gen luo
674805864fedd3964253eae1
Yuqin Yang
admin_assigned
2024-11-28T13:12:28.350Z
64672f4aa9b4610868a0407e
/avatars/2e964e7edd2d1ebaa2b8597992090925.svg
Yuqin Yang
user
CRIS-Yang
674805864fedd3964253eae2
Yuda Xiong
674805864fedd3964253eae3
Yihao Chen
674805864fedd3964253eae4
Zhaoyang Zeng
admin_assigned
2024-11-28T12:29:31.995Z
669f0abdc13da2bbe268c5c3
/avatars/87961bfd01c53c420ae13133d643f22c.svg
Zhaoyang Zeng
user
zengzhaoyang
674805864fedd3964253eae5
Tianhe Ren
674805864fedd3964253eae6
Lei Zhang
ChatRex: Taming Multimodal LLM for Joint Perception and Understanding
Perception and understanding are two pillars of computer vision. While
multimodal large language models (MLLM) have demonstrated remarkable visual
understanding capabilities, they arguably lack accurate perception abilities,
e.g. the stage-of-the-art model Qwen2-VL only achieves a 43.9 recall rate on
the COCO dataset, limiting many tasks requiring the combination of perception
and understanding. In this work, we aim to bridge this perception gap from both
model designing and data development perspectives. We first introduce ChatRex,
an MLLM with a decoupled perception design. Instead of having the LLM directly
predict box coordinates, we feed the output boxes from a universal proposal
network into the LLM, allowing it to output the corresponding box indices to
represent its detection results, turning the regression task into a
retrieval-based task that LLM handles more proficiently. From the data
perspective, we build a fully automated data engine and construct the
Rexverse-2M dataset which possesses multiple granularities to support the joint
training of perception and understanding. After standard two-stage training,
ChatRex demonstrates strong perception capabilities while preserving multimodal
understanding performance. The combination of these two capabilities
simultaneously unlocks many attractive applications, demonstrating the
complementary roles of both perception and understanding in MLLM. Code is
available at https://github.com/IDEA-Research/ChatRex.
ChatRex, a multimodal large language model with decoupled perception and a new dataset, enhances both perception and understanding capabilities.
multimodal large language models decoupled perception ChatRex universal proposal network box coordinates retrieval-based task fully automated data engine Rexverse-2M dataset standard two-stage training