title: FUSION: Fully Integration of Vision-Language Representations for Deep
    Cross-Modal Understanding
summary: We introduce FUSION, a family of multimodal large language models (MLLMs)
  with a fully vision-language alignment and integration paradigm. Unlike
  existing methods that primarily rely on late-stage modality interaction during
  LLM decoding, our approach achieves deep, dynamic integration throughout the
  entire processing pipeline. To this end, we propose Text-Guided Unified Vision
  Encoding, incorporating textual information in vision encoding to achieve
  pixel-level integration. We further design Context-Aware Recursive Alignment
  Decoding that recursively aggregates visual features conditioned on textual
  context during decoding, enabling fine-grained, question-level semantic
  integration. To guide feature mapping and mitigate modality discrepancies, we
  develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
  Synthesized Language-Driven Question-Answer (QA) dataset through a new data
  synthesis method, prioritizing high-quality QA pairs to optimize text-guided
  feature integration. Building on these foundations, we train FUSION at two
  scales-3B, 8B-and demonstrate that our full-modality integration approach
  significantly outperforms existing methods with only 630 vision tokens.
  Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
  benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
  to 300 vision tokens. Our ablation studies show that FUSION outperforms
  LLaVA-NeXT on over half of the benchmarks under same configuration without
  dynamic resolution, highlighting the effectiveness of our approach. We release
  our code, model weights, and dataset. https://github.com/starriver030515/FUSION
paper_authors._id: 67fdb5f1913c97aa32f130bd
paper_authors.name: Zheng Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:57.129Z
paper_authors._id: 6625ef13605f46d05c1d0031
paper_authors.avatarUrl: /avatars/22f201dca35e43013cb593884516e96c.svg
paper_authors.fullname: Zheng Liu
paper_authors.type: user
paper_authors.user: starriver030515
paper_authors._id: 67fdb5f1913c97aa32f130be
paper_authors.name: Mengjie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:16:15.647Z
paper_authors._id: 672dbf1bec84b9c33412488f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/V6o6K28ydpkaS9XFA4Ac3.png
paper_authors.fullname: Mengjie Liu
paper_authors.type: user
paper_authors.user: Balalauuoo
paper_authors._id: 67fdb5f1913c97aa32f130bf
paper_authors.name: Jingzhou Chen
paper_authors._id: 67fdb5f1913c97aa32f130c0
paper_authors.name: Jingwei Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:16:53.642Z
paper_authors._id: 672da19bb2f2dc21e176b0de
paper_authors.avatarUrl: /avatars/a942ad1c467b865cb9530927fe13f2b7.svg
paper_authors.fullname: Jingwei Xu
paper_authors.type: user
paper_authors.user: jingwei-xu-00
paper_authors._id: 67fdb5f1913c97aa32f130c1
paper_authors.name: Bin Cui
paper_authors._id: 67fdb5f1913c97aa32f130c2
paper_authors.name: Conghui He
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:17:01.171Z
paper_authors._id: 63f9fca8d4349b157a109eec
paper_authors.avatarUrl: /avatars/fa1f2ae7972d7cde99dab178136ccbb0.svg
paper_authors.fullname: Conghui He
paper_authors.type: user
paper_authors.user: conghui
paper_authors._id: 67fdb5f1913c97aa32f130c3
paper_authors.name: Wentao Zhang
paper_title: FUSION: Fully Integration of Vision-Language Representations for Deep
    Cross-Modal Understanding
paper_summary: We introduce FUSION, a family of multimodal large language models (MLLMs)
  with a fully vision-language alignment and integration paradigm. Unlike
  existing methods that primarily rely on late-stage modality interaction during
  LLM decoding, our approach achieves deep, dynamic integration throughout the
  entire processing pipeline. To this end, we propose Text-Guided Unified Vision
  Encoding, incorporating textual information in vision encoding to achieve
  pixel-level integration. We further design Context-Aware Recursive Alignment
  Decoding that recursively aggregates visual features conditioned on textual
  context during decoding, enabling fine-grained, question-level semantic
  integration. To guide feature mapping and mitigate modality discrepancies, we
  develop Dual-Supervised Semantic Mapping Loss. Additionally, we construct a
  Synthesized Language-Driven Question-Answer (QA) dataset through a new data
  synthesis method, prioritizing high-quality QA pairs to optimize text-guided
  feature integration. Building on these foundations, we train FUSION at two
  scales-3B, 8B-and demonstrate that our full-modality integration approach
  significantly outperforms existing methods with only 630 vision tokens.
  Notably, FUSION 3B surpasses Cambrian-1 8B and Florence-VL 8B on most
  benchmarks. FUSION 3B continues to outperform Cambrian-1 8B even when limited
  to 300 vision tokens. Our ablation studies show that FUSION outperforms
  LLaVA-NeXT on over half of the benchmarks under same configuration without
  dynamic resolution, highlighting the effectiveness of our approach. We release
  our code, model weights, and dataset. https://github.com/starriver030515/FUSION
paper_ai_summary: FUSION, a multimodal large language model, integrates vision and language through deep, pixel-level and question-level integration, achieving superior performance compared to existing methods.
paper_ai_keywords: multimodal large language models MLLMs vision-language alignment Text-Guided Unified Vision Encoding Context-Aware Recursive Alignment Decoding Dual-Supervised Semantic Mapping Loss Synthesized Language-Driven Question-Answer QA dataset full-modality integration feature integration seamless vision-language interaction FUSION 3B FUSION 8B Cambrian-1 8B Florence-VL 8B LLaVA-NeXT dynamic resolution