title: Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling
    and Visual-Language Co-Referring
summary: Large Vision Language Models have achieved fine-grained object perception,
  but the limitation of image resolution remains a significant obstacle to
  surpass the performance of task-specific experts in complex and dense
  scenarios. Such limitation further restricts the model's potential to achieve
  nuanced visual and language referring in domains such as GUI Agents, Counting
  and \etc. To address this issue, we introduce a unified high-resolution
  generalist model, Griffon v2, enabling flexible object referring with visual
  and textual prompts. To efficiently scaling up image resolution, we design a
  simple and lightweight down-sampling projector to overcome the input tokens
  constraint in Large Language Models. This design inherently preserves the
  complete contexts and fine details, and significantly improves multimodal
  perception ability especially for small objects. Building upon this, we further
  equip the model with visual-language co-referring capabilities through a
  plug-and-play visual tokenizer. It enables user-friendly interaction with
  flexible target images, free-form texts and even coordinates. Experiments
  demonstrate that Griffon v2 can localize any objects of interest with visual
  and textual referring, achieve state-of-the-art performance on REC, phrase
  grounding, and REG tasks, and outperform expert models in object detection and
  object counting. Data, codes and models will be released at
  https://github.com/jefferyZhan/Griffon.
paper_authors._id: 65f3c2aedb765cc2d16988ac
paper_authors.name: Yufei Zhan
paper_authors._id: 65f3c2aedb765cc2d16988ad
paper_authors.name: Yousong Zhu
paper_authors._id: 65f3c2aedb765cc2d16988ae
paper_authors.name: Hongyin Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-15T13:18:34.607Z
paper_authors._id: 65d6d107b06abf924b23bd65
paper_authors.avatarUrl: /avatars/00835205ad84f2d3682581053a1d1c64.svg
paper_authors.fullname: hongyin zhao
paper_authors.type: user
paper_authors.user: Zahooying
paper_authors._id: 65f3c2aedb765cc2d16988af
paper_authors.name: Fan Yang
paper_authors._id: 65f3c2aedb765cc2d16988b0
paper_authors.name: Ming Tang
paper_authors._id: 65f3c2aedb765cc2d16988b1
paper_authors.name: Jinqiao Wang
paper_title: Griffon v2: Advancing Multimodal Perception with High-Resolution Scaling
    and Visual-Language Co-Referring
paper_summary: Large Vision Language Models have achieved fine-grained object perception,
  but the limitation of image resolution remains a significant obstacle to
  surpass the performance of task-specific experts in complex and dense
  scenarios. Such limitation further restricts the model's potential to achieve
  nuanced visual and language referring in domains such as GUI Agents, Counting
  and \etc. To address this issue, we introduce a unified high-resolution
  generalist model, Griffon v2, enabling flexible object referring with visual
  and textual prompts. To efficiently scaling up image resolution, we design a
  simple and lightweight down-sampling projector to overcome the input tokens
  constraint in Large Language Models. This design inherently preserves the
  complete contexts and fine details, and significantly improves multimodal
  perception ability especially for small objects. Building upon this, we further
  equip the model with visual-language co-referring capabilities through a
  plug-and-play visual tokenizer. It enables user-friendly interaction with
  flexible target images, free-form texts and even coordinates. Experiments
  demonstrate that Griffon v2 can localize any objects of interest with visual
  and textual referring, achieve state-of-the-art performance on REC, phrase
  grounding, and REG tasks, and outperform expert models in object detection and
  object counting. Data, codes and models will be released at
  https://github.com/jefferyZhan/Griffon.
paper_ai_summary: Griffon v2, a unified high-resolution model with a down-sampling projector and visual tokenizer, achieves state-of-the-art performance in object detection, counting, and referring tasks by overcoming image resolution and multimodal perception limitations.
paper_ai_keywords: down-sampling projector visual tokenizer multimodal perception REC phrase grounding REG tasks