title: Ferret-v2: An Improved Baseline for Referring and Grounding with Large
    Language Models
summary: While Ferret seamlessly integrates regional understanding into the Large
  Language Model (LLM) to facilitate its referring and grounding capability, it
  poses certain limitations: constrained by the pre-trained fixed visual encoder
  and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
  a significant upgrade to Ferret, with three key designs. (1) Any resolution
  grounding and referring: A flexible approach that effortlessly handles higher
  image resolution, improving the model's ability to process and understand
  images in greater detail. (2) Multi-granularity visual encoding: By integrating
  the additional DINOv2 encoder, the model learns better and diverse underlying
  contexts for global and fine-grained visual information. (3) A three-stage
  training paradigm: Besides image-caption alignment, an additional stage is
  proposed for high-resolution dense alignment before the final instruction
  tuning. Experiments show that Ferret-v2 provides substantial improvements over
  Ferret and other state-of-the-art methods, thanks to its high-resolution
  scaling and fine-grained visual processing.
paper_authors._id: 661891f2cbf0f99bc8116db2
paper_authors.name: Haotian Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-04-12T03:05:16.871Z
paper_authors._id: 631516348d85ad332fa47b2c
paper_authors.avatarUrl: /avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
paper_authors.fullname: Haotian Zhang
paper_authors.type: user
paper_authors.user: haotiz
paper_authors._id: 661891f2cbf0f99bc8116db3
paper_authors.name: Haoxuan You
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:12:21.538Z
paper_authors._id: 65284ba3e3b3844f5e487a0b
paper_authors.avatarUrl: /avatars/fce12a233bddb611d441f25a2ab17725.svg
paper_authors.fullname: Haoxuan You
paper_authors.type: user
paper_authors.user: HaoxuanYou
paper_authors._id: 661891f2cbf0f99bc8116db4
paper_authors.name: Philipp Dufter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:12:27.999Z
paper_authors._id: 6079980b88160e14e4e2e49d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
paper_authors.fullname: Philipp Dufter
paper_authors.type: user
paper_authors.user: pdufter
paper_authors._id: 661891f2cbf0f99bc8116db5
paper_authors.name: Bowen Zhang
paper_authors._id: 661891f2cbf0f99bc8116db6
paper_authors.name: Chen Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:24:05.307Z
paper_authors._id: 65409e53d07194a81ecdd427
paper_authors.avatarUrl: /avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
paper_authors.fullname: Chen Chen
paper_authors.type: user
paper_authors.user: ultra7chen
paper_authors._id: 661891f2cbf0f99bc8116db7
paper_authors.name: Hong-You Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:12:41.938Z
paper_authors._id: 64b720891a4d97b5d7f055b8
paper_authors.avatarUrl: /avatars/7247a4f3a433a58fe934f245d9a757b2.svg
paper_authors.fullname: Hong-You Chen
paper_authors.type: user
paper_authors.user: Cbbjames
paper_authors._id: 661891f2cbf0f99bc8116db8
paper_authors.name: Tsu-Jui Fu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:12:48.180Z
paper_authors._id: 638cb2efc8912be69c149a46
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
paper_authors.fullname: Tsu-Jui Fu
paper_authors.type: user
paper_authors.user: tsujuifu
paper_authors._id: 661891f2cbf0f99bc8116db9
paper_authors.name: William Yang Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:13:08.669Z
paper_authors._id: 6476607efb22e3b77f3f1452
paper_authors.avatarUrl: /avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
paper_authors.fullname: William Yang Wang
paper_authors.type: user
paper_authors.user: wangwilliamyang
paper_authors._id: 661891f2cbf0f99bc8116dba
paper_authors.name: Shih-Fu Chang
paper_authors._id: 661891f2cbf0f99bc8116dbb
paper_authors.name: Zhe Gan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:13:32.079Z
paper_authors._id: 644022db21ec7c165d848b6e
paper_authors.avatarUrl: /avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
paper_authors.fullname: Zhe Gan
paper_authors.type: user
paper_authors.user: zhegan27
paper_authors._id: 661891f2cbf0f99bc8116dbc
paper_authors.name: Yinfei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T12:13:38.770Z
paper_authors._id: 64b762568c632fbca942a405
paper_authors.avatarUrl: /avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
paper_authors.fullname: Yinfei Yang
paper_authors.type: user
paper_authors.user: yinfeiy
paper_title: Ferret-v2: An Improved Baseline for Referring and Grounding with Large
    Language Models
paper_summary: While Ferret seamlessly integrates regional understanding into the Large
  Language Model (LLM) to facilitate its referring and grounding capability, it
  poses certain limitations: constrained by the pre-trained fixed visual encoder
  and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
  a significant upgrade to Ferret, with three key designs. (1) Any resolution
  grounding and referring: A flexible approach that effortlessly handles higher
  image resolution, improving the model's ability to process and understand
  images in greater detail. (2) Multi-granularity visual encoding: By integrating
  the additional DINOv2 encoder, the model learns better and diverse underlying
  contexts for global and fine-grained visual information. (3) A three-stage
  training paradigm: Besides image-caption alignment, an additional stage is
  proposed for high-resolution dense alignment before the final instruction
  tuning. Experiments show that Ferret-v2 provides substantial improvements over
  Ferret and other state-of-the-art methods, thanks to its high-resolution
  scaling and fine-grained visual processing.
paper_ai_summary: Ferret-v2 improves on Ferret by incorporating flexible high-resolution grounding, multi-granularity visual encoding with DINOv2, and a three-stage training paradigm, leading to enhanced visual understanding and performance.
paper_ai_keywords: regional understanding Large Language Model (LLM) visual encoder DINOv2 multi-granularity visual encoding high-resolution grounding three-stage training paradigm image-caption alignment high-resolution dense alignment