title: OneLLM: One Framework to Align All Modalities with Language
summary: Multimodal large language models (MLLMs) have gained significant attention
  due to their strong multimodal understanding capability. However, existing
  works rely heavily on modality-specific encoders, which usually differ in
  architecture and are limited to common modalities. In this paper, we present
  OneLLM, an MLLM that aligns eight modalities to language using a unified
  framework. We achieve this through a unified multimodal encoder and a
  progressive multimodal alignment pipeline. In detail, we first train an image
  projection module to connect a vision encoder with LLM. Then, we build a
  universal projection module (UPM) by mixing multiple image projection modules
  and dynamic routing. Finally, we progressively align more modalities to LLM
  with the UPM. To fully leverage the potential of OneLLM in following
  instructions, we also curated a comprehensive multimodal instruction dataset,
  including 2M items from image, audio, video, point cloud, depth/normal map, IMU
  and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks,
  encompassing tasks such as multimodal captioning, question answering and
  reasoning, where it delivers excellent performance. Code, data, model and
  online demo are available at https://github.com/csuhan/OneLLM
paper_authors._id: 65712870c8018fe6408b5376
paper_authors.name: Jiaming Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-07T08:29:55.390Z
paper_authors._id: 62318c0386753f5f41d0e261
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62318c0386753f5f41d0e261/xO_5PvOf7lXhQPnQLcmnq.jpeg
paper_authors.fullname: Jiaming Han
paper_authors.type: user
paper_authors.user: csuhan
paper_authors._id: 65712870c8018fe6408b5377
paper_authors.name: Kaixiong Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:38:07.316Z
paper_authors._id: 642e427f6748dd4f8eeb2f38
paper_authors.avatarUrl: /avatars/07158ff6aa1803c846403594c5d55a34.svg
paper_authors.fullname: Kaixiong Gong
paper_authors.type: user
paper_authors.user: kxgong
paper_authors._id: 65712870c8018fe6408b5378
paper_authors.name: Yiyuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-29T08:28:41.137Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 65712870c8018fe6408b5379
paper_authors.name: Jiaqi Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:39:41.220Z
paper_authors._id: 64b4eec4faa3181a5eab9c46
paper_authors.avatarUrl: /avatars/bcc9bf5cbf67546ad2b4c9ec8b96ac96.svg
paper_authors.fullname: Jiaqi Wang
paper_authors.type: user
paper_authors.user: myownskyW7
paper_authors._id: 65712870c8018fe6408b537a
paper_authors.name: Kaipeng Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:39:49.235Z
paper_authors._id: 63527f4e7d071f23d085ad45
paper_authors.avatarUrl: /avatars/99a51adef5673b3ac1a8c02eb47759c4.svg
paper_authors.fullname: KAIPENG ZHANG
paper_authors.type: user
paper_authors.user: kpzhang
paper_authors._id: 65712870c8018fe6408b537b
paper_authors.name: Dahua Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-07T11:39:55.752Z
paper_authors._id: 636317ed80c1a705a6eff396
paper_authors.avatarUrl: /avatars/3db090e101b916d9256d0d3e043db71d.svg
paper_authors.fullname: Dahua Lin
paper_authors.type: user
paper_authors.user: lindahua
paper_authors._id: 65712870c8018fe6408b537c
paper_authors.name: Yu Qiao
paper_authors._id: 65712870c8018fe6408b537d
paper_authors.name: Peng Gao
paper_authors._id: 65712870c8018fe6408b537e
paper_authors.name: Xiangyu Yue
paper_title: OneLLM: One Framework to Align All Modalities with Language
paper_summary: Multimodal large language models (MLLMs) have gained significant attention
  due to their strong multimodal understanding capability. However, existing
  works rely heavily on modality-specific encoders, which usually differ in
  architecture and are limited to common modalities. In this paper, we present
  OneLLM, an MLLM that aligns eight modalities to language using a unified
  framework. We achieve this through a unified multimodal encoder and a
  progressive multimodal alignment pipeline. In detail, we first train an image
  projection module to connect a vision encoder with LLM. Then, we build a
  universal projection module (UPM) by mixing multiple image projection modules
  and dynamic routing. Finally, we progressively align more modalities to LLM
  with the UPM. To fully leverage the potential of OneLLM in following
  instructions, we also curated a comprehensive multimodal instruction dataset,
  including 2M items from image, audio, video, point cloud, depth/normal map, IMU
  and fMRI brain activity. OneLLM is evaluated on 25 diverse benchmarks,
  encompassing tasks such as multimodal captioning, question answering and
  reasoning, where it delivers excellent performance. Code, data, model and
  online demo are available at https://github.com/csuhan/OneLLM
paper_ai_keywords: multimodal large language models (MLLMs) unified framework unified multimodal encoder progressive multimodal alignment pipeline image projection module vision encoder universal projection module (UPM) dynamic routing multimodal instruction dataset multimodal captioning question answering reasoning