title: Florence-VL: Enhancing Vision-Language Models with Generative Vision
    Encoder and Depth-Breadth Fusion
summary: We present Florence-VL, a new family of multimodal large language models
  (MLLMs) with enriched visual representations produced by Florence-2, a
  generative vision foundation model. Unlike the widely used CLIP-style vision
  transformer trained by contrastive learning, Florence-2 can capture different
  levels and aspects of visual features, which are more versatile to be adapted
  to diverse downstream tasks. We propose a novel feature-fusion architecture and
  an innovative training recipe that effectively integrates Florence-2's visual
  features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
  propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
  from different depths and under multiple prompts. Our model training is
  composed of end-to-end pretraining of the whole model followed by finetuning of
  the projection layer and the LLM, on a carefully designed recipe of diverse
  open-source datasets that include high-quality image captions and
  instruction-tuning pairs. Our quantitative analysis and visualization of
  Florence-VL's visual features show its advantages over popular vision encoders
  on vision-language alignment, where the enriched depth and breath play
  important roles. Florence-VL achieves significant improvements over existing
  state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
  covering general VQA, perception, hallucination, OCR, Chart,
  knowledge-intensive understanding, etc. To facilitate future research, our
  models and the complete training recipe are open-sourced.
  https://github.com/JiuhaiChen/Florence-VL
paper_authors._id: 67526d9bba72c8bc0779edcf
paper_authors.name: Jiuhai Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:18:45.149Z
paper_authors._id: 6393847e3e30234ae798b7be
paper_authors.avatarUrl: /avatars/daeb8c37dff4432d837a69b87c196521.svg
paper_authors.fullname: JiuhaiChen
paper_authors.type: user
paper_authors.user: jiuhai
paper_authors._id: 67526d9bba72c8bc0779edd0
paper_authors.name: Jianwei Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:18:34.697Z
paper_authors._id: 6125df7f25027fb1ea9c7a41
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1629871954341-noauth.jpeg
paper_authors.fullname: Jianwei Yang
paper_authors.type: user
paper_authors.user: jw2yang
paper_authors._id: 67526d9bba72c8bc0779edd1
paper_authors.name: Haiping Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:18:17.565Z
paper_authors._id: 6634347b6cab8677737be59b
paper_authors.avatarUrl: /avatars/1deebaa2c63309e37df62be4c6521443.svg
paper_authors.fullname: Haiping Wu
paper_authors.type: user
paper_authors.user: haipingwu
paper_authors._id: 67526d9bba72c8bc0779edd2
paper_authors.name: Dianqi Li
paper_authors._id: 67526d9bba72c8bc0779edd3
paper_authors.name: Jianfeng Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-06T17:17:57.025Z
paper_authors._id: 641904caf9d6f1d772ec7af7
paper_authors.avatarUrl: /avatars/4a63eac71eb30f70b1a0e9d4708f26c1.svg
paper_authors.fullname: Jianfeng Gao
paper_authors.type: user
paper_authors.user: wyngjf
paper_authors._id: 67526d9bba72c8bc0779edd4
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-06T09:09:02.653Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_authors._id: 67526d9bba72c8bc0779edd5
paper_authors.name: Bin Xiao
paper_title: Florence-VL: Enhancing Vision-Language Models with Generative Vision
    Encoder and Depth-Breadth Fusion
paper_summary: We present Florence-VL, a new family of multimodal large language models
  (MLLMs) with enriched visual representations produced by Florence-2, a
  generative vision foundation model. Unlike the widely used CLIP-style vision
  transformer trained by contrastive learning, Florence-2 can capture different
  levels and aspects of visual features, which are more versatile to be adapted
  to diverse downstream tasks. We propose a novel feature-fusion architecture and
  an innovative training recipe that effectively integrates Florence-2's visual
  features into pretrained LLMs, such as Phi 3.5 and LLama 3. In particular, we
  propose "depth-breath fusion (DBFusion)" to fuse the visual features extracted
  from different depths and under multiple prompts. Our model training is
  composed of end-to-end pretraining of the whole model followed by finetuning of
  the projection layer and the LLM, on a carefully designed recipe of diverse
  open-source datasets that include high-quality image captions and
  instruction-tuning pairs. Our quantitative analysis and visualization of
  Florence-VL's visual features show its advantages over popular vision encoders
  on vision-language alignment, where the enriched depth and breath play
  important roles. Florence-VL achieves significant improvements over existing
  state-of-the-art MLLMs across various multi-modal and vision-centric benchmarks
  covering general VQA, perception, hallucination, OCR, Chart,
  knowledge-intensive understanding, etc. To facilitate future research, our
  models and the complete training recipe are open-sourced.
  https://github.com/JiuhaiChen/Florence-VL
paper_ai_keywords: multimodal large language models (MLLMs) visual representations Florence-2 generative vision foundation model CLIP-style vision transformer contrastive learning visual features feature-fusion architecture depth-breath fusion (DBFusion) pretraining finetuning projection layer diverse open-source datasets high-quality image captions instruction-tuning pairs vision-language alignment general VQA perception hallucination OCR Chart knowledge-intensive understanding