Ferret-v2: An Improved Baseline for Referring and Grounding with Large
  Language Models
While Ferret seamlessly integrates regional understanding into the Large
Language Model (LLM) to facilitate its referring and grounding capability, it
poses certain limitations: constrained by the pre-trained fixed visual encoder
and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
a significant upgrade to Ferret, with three key designs. (1) Any resolution
grounding and referring: A flexible approach that effortlessly handles higher
image resolution, improving the model's ability to process and understand
images in greater detail. (2) Multi-granularity visual encoding: By integrating
the additional DINOv2 encoder, the model learns better and diverse underlying
contexts for global and fine-grained visual information. (3) A three-stage
training paradigm: Besides image-caption alignment, an additional stage is
proposed for high-resolution dense alignment before the final instruction
tuning. Experiments show that Ferret-v2 provides substantial improvements over
Ferret and other state-of-the-art methods, thanks to its high-resolution
scaling and fine-grained visual processing.
661891f2cbf0f99bc8116db2
Haotian Zhang
claimed_verified
2024-04-12T03:05:16.871Z
631516348d85ad332fa47b2c
/avatars/100f5ae3cf3c52faaecdaecd5d8f2881.svg
Haotian Zhang
user
haotiz
661891f2cbf0f99bc8116db3
Haoxuan You
admin_assigned
2024-04-18T12:12:21.538Z
65284ba3e3b3844f5e487a0b
/avatars/fce12a233bddb611d441f25a2ab17725.svg
Haoxuan You
user
HaoxuanYou
661891f2cbf0f99bc8116db4
Philipp Dufter
admin_assigned
2024-04-18T12:12:27.999Z
6079980b88160e14e4e2e49d
https://cdn-avatars.huggingface.co/v1/production/uploads/6079980b88160e14e4e2e49d/pHROmoIP5xRWNrVufajyf.jpeg
Philipp Dufter
user
pdufter
661891f2cbf0f99bc8116db5
Bowen Zhang
661891f2cbf0f99bc8116db6
Chen Chen
claimed_verified
2025-03-16T21:24:05.307Z
65409e53d07194a81ecdd427
/avatars/73a88ffe7f78f05b6fc8eb57e9f1e587.svg
Chen Chen
user
ultra7chen
661891f2cbf0f99bc8116db7
Hong-You Chen
admin_assigned
2024-04-18T12:12:41.938Z
64b720891a4d97b5d7f055b8
/avatars/7247a4f3a433a58fe934f245d9a757b2.svg
Hong-You Chen
user
Cbbjames
661891f2cbf0f99bc8116db8
Tsu-Jui Fu
admin_assigned
2024-04-18T12:12:48.180Z
638cb2efc8912be69c149a46
https://cdn-avatars.huggingface.co/v1/production/uploads/638cb2efc8912be69c149a46/qxMNOWoSUG525DgNPpHcK.jpeg
Tsu-Jui Fu
user
tsujuifu
661891f2cbf0f99bc8116db9
William Yang Wang
admin_assigned
2024-04-18T12:13:08.669Z
6476607efb22e3b77f3f1452
/avatars/22f40c81e6b6e09bdb66a5afd978f98b.svg
William Yang Wang
user
wangwilliamyang
661891f2cbf0f99bc8116dba
Shih-Fu Chang
661891f2cbf0f99bc8116dbb
Zhe Gan
admin_assigned
2024-04-18T12:13:32.079Z
644022db21ec7c165d848b6e
/avatars/d9ed37ec77f2682878b2976a04d2fcd8.svg
Zhe Gan
user
zhegan27
661891f2cbf0f99bc8116dbc
Yinfei Yang
admin_assigned
2024-04-18T12:13:38.770Z
64b762568c632fbca942a405
/avatars/1eb737ec169967872f1ebf5ff29f1e6b.svg
Yinfei Yang
user
yinfeiy
Ferret-v2: An Improved Baseline for Referring and Grounding with Large
  Language Models
While Ferret seamlessly integrates regional understanding into the Large
Language Model (LLM) to facilitate its referring and grounding capability, it
poses certain limitations: constrained by the pre-trained fixed visual encoder
and failed to perform well on broader tasks. In this work, we unveil Ferret-v2,
a significant upgrade to Ferret, with three key designs. (1) Any resolution
grounding and referring: A flexible approach that effortlessly handles higher
image resolution, improving the model's ability to process and understand
images in greater detail. (2) Multi-granularity visual encoding: By integrating
the additional DINOv2 encoder, the model learns better and diverse underlying
contexts for global and fine-grained visual information. (3) A three-stage
training paradigm: Besides image-caption alignment, an additional stage is
proposed for high-resolution dense alignment before the final instruction
tuning. Experiments show that Ferret-v2 provides substantial improvements over
Ferret and other state-of-the-art methods, thanks to its high-resolution
scaling and fine-grained visual processing.
Ferret-v2 improves on Ferret by incorporating flexible high-resolution grounding, multi-granularity visual encoding with DINOv2, and a three-stage training paradigm, leading to enhanced visual understanding and performance.
regional understanding Large Language Model (LLM) visual encoder DINOv2 multi-granularity visual encoding high-resolution grounding three-stage training paradigm image-caption alignment high-resolution dense alignment