Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling
  and Visual-Language Co-Referring
Large Vision Language Models have achieved fine-grained object perception,
but the limitation of image resolution remains a significant obstacle to
surpass the performance of task-specific experts in complex and dense
scenarios. Such limitation further restricts the model's potential to achieve
nuanced visual and language referring in domains such as GUI Agents, Counting
and \etc. To address this issue, we introduce a unified high-resolution
generalist model, Griffon v2, enabling flexible object referring with visual
and textual prompts. To efficiently scaling up image resolution, we design a
simple and lightweight down-sampling projector to overcome the input tokens
constraint in Large Language Models. This design inherently preserves the
complete contexts and fine details, and significantly improves multimodal
perception ability especially for small objects. Building upon this, we further
equip the model with visual-language co-referring capabilities through a
plug-and-play visual tokenizer. It enables user-friendly interaction with
flexible target images, free-form texts and even coordinates. Experiments
demonstrate that Griffon v2 can localize any objects of interest with visual
and textual referring, achieve state-of-the-art performance on REC, phrase
grounding, and REG tasks, and outperform expert models in object detection and
object counting. Data, codes and models will be released at
https://github.com/jefferyZhan/Griffon.
65f3c2aedb765cc2d16988ac
Yufei Zhan
65f3c2aedb765cc2d16988ad
Yousong Zhu
65f3c2aedb765cc2d16988ae
Hongyin Zhao
admin_assigned
2024-03-15T13:18:34.607Z
65d6d107b06abf924b23bd65
/avatars/00835205ad84f2d3682581053a1d1c64.svg
hongyin zhao
user
Zahooying
65f3c2aedb765cc2d16988af
Fan Yang
65f3c2aedb765cc2d16988b0
Ming Tang
65f3c2aedb765cc2d16988b1
Jinqiao Wang
Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling
  and Visual-Language Co-Referring
Large Vision Language Models have achieved fine-grained object perception,
but the limitation of image resolution remains a significant obstacle to
surpass the performance of task-specific experts in complex and dense
scenarios. Such limitation further restricts the model's potential to achieve
nuanced visual and language referring in domains such as GUI Agents, Counting
and \etc. To address this issue, we introduce a unified high-resolution
generalist model, Griffon v2, enabling flexible object referring with visual
and textual prompts. To efficiently scaling up image resolution, we design a
simple and lightweight down-sampling projector to overcome the input tokens
constraint in Large Language Models. This design inherently preserves the
complete contexts and fine details, and significantly improves multimodal
perception ability especially for small objects. Building upon this, we further
equip the model with visual-language co-referring capabilities through a
plug-and-play visual tokenizer. It enables user-friendly interaction with
flexible target images, free-form texts and even coordinates. Experiments
demonstrate that Griffon v2 can localize any objects of interest with visual
and textual referring, achieve state-of-the-art performance on REC, phrase
grounding, and REG tasks, and outperform expert models in object detection and
object counting. Data, codes and models will be released at
https://github.com/jefferyZhan/Griffon.
Griffon v2, a unified high-resolution model with a down-sampling projector and visual tokenizer, achieves state-of-the-art performance in object detection, counting, and referring tasks by overcoming image resolution and multimodal perception limitations.
down-sampling projector visual tokenizer multimodal perception REC phrase grounding REG tasks