title: X-Fusion: Introducing New Modality to Frozen Large Language Models
summary: We propose X-Fusion, a framework that extends pretrained Large Language
  Models (LLMs) for multimodal tasks while preserving their language
  capabilities. X-Fusion employs a dual-tower design with modality-specific
  weights, keeping the LLM's parameters frozen while integrating vision-specific
  information for both understanding and generation. Our experiments demonstrate
  that X-Fusion consistently outperforms alternative architectures on both
  image-to-text and text-to-image tasks. We find that incorporating
  understanding-focused data improves generation quality, reducing image data
  noise enhances overall performance, and feature alignment accelerates
  convergence for smaller models but has minimal impact on larger ones. Our
  findings provide valuable insights into building efficient unified multimodal
  models.
paper_authors._id: 6811c55384adfa26b82abd76
paper_authors.name: Sicheng Mo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:22:58.085Z
paper_authors._id: 637c94d3f219c71f93eda9ad
paper_authors.avatarUrl: /avatars/6dae0c30755196ccc0a5a06b3981c47f.svg
paper_authors.fullname: Sicheng Mo
paper_authors.type: user
paper_authors.user: Sichengmo
paper_authors._id: 6811c55384adfa26b82abd77
paper_authors.name: Thao Nguyen
paper_authors._id: 6811c55384adfa26b82abd78
paper_authors.name: Xun Huang
paper_authors._id: 6811c55384adfa26b82abd79
paper_authors.name: Siddharth Srinivasan Iyer
paper_authors._id: 6811c55384adfa26b82abd7a
paper_authors.name: Yijun Li
paper_authors._id: 6811c55384adfa26b82abd7b
paper_authors.name: Yuchen Liu
paper_authors._id: 6811c55384adfa26b82abd7c
paper_authors.name: Abhishek Tandon
paper_authors._id: 6811c55384adfa26b82abd7d
paper_authors.name: Eli Shechtman
paper_authors._id: 6811c55384adfa26b82abd7e
paper_authors.name: Krishna Kumar Singh
paper_authors._id: 6811c55384adfa26b82abd7f
paper_authors.name: Yong Jae Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:23:52.478Z
paper_authors._id: 649f41ee70a478f8b36b2984
paper_authors.avatarUrl: /avatars/1c9a76717a450ac4aeb62a1e823d2e4a.svg
paper_authors.fullname: Yong Jae Lee
paper_authors.type: user
paper_authors.user: yjlee0222
paper_authors._id: 6811c55384adfa26b82abd80
paper_authors.name: Bolei Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-30T12:23:58.680Z
paper_authors._id: 64aedf11c693a19547999a59
paper_authors.avatarUrl: /avatars/83a54b7c217168c91cea15dd000f4af2.svg
paper_authors.fullname: Bolei Zhou
paper_authors.type: user
paper_authors.user: zhoubolei
paper_authors._id: 6811c55384adfa26b82abd81
paper_authors.name: Yuheng Li
paper_title: X-Fusion: Introducing New Modality to Frozen Large Language Models
paper_summary: We propose X-Fusion, a framework that extends pretrained Large Language
  Models (LLMs) for multimodal tasks while preserving their language
  capabilities. X-Fusion employs a dual-tower design with modality-specific
  weights, keeping the LLM's parameters frozen while integrating vision-specific
  information for both understanding and generation. Our experiments demonstrate
  that X-Fusion consistently outperforms alternative architectures on both
  image-to-text and text-to-image tasks. We find that incorporating
  understanding-focused data improves generation quality, reducing image data
  noise enhances overall performance, and feature alignment accelerates
  convergence for smaller models but has minimal impact on larger ones. Our
  findings provide valuable insights into building efficient unified multimodal
  models.
paper_ai_summary: X-Fusion enhances pretrained LLMs for multimodal tasks by integrating vision-specific information through a dual-tower design and freezes the LLM's parameters for consistency in understanding and generation.
paper_ai_keywords: Large Language Models (LLMs) multimodal tasks dual-tower design modality-specific weights vision-specific information image-to-text text-to-image understanding-focused data image data noise feature alignment