title: X-LLM: Bootstrapping Advanced Large Language Models by Treating
    Multi-Modalities as Foreign Languages
summary: Large language models (LLMs) have demonstrated remarkable language abilities.
  GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities
  beyond previous visual language models. We attribute this to the use of more
  advanced LLMs compared with previous multimodal models. Unfortunately, the
  model architecture and training strategies of GPT-4 are unknown. To endow LLMs
  with multimodal capabilities, we propose X-LLM, which converts Multi-modalities
  (images, speech, videos) into foreign languages using X2L interfaces and inputs
  them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple
  frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''
  denotes multi-modalities such as image, speech, and videos, and ``L'' denotes
  languages. X-LLM's training consists of three stages: (1) Converting Multimodal
  Information: The first stage trains each X2L interface to align with its
  respective single-modal encoder separately to convert multimodal information
  into languages. (2) Aligning X2L representations with the LLM: single-modal
  encoders are aligned with the LLM through X2L interfaces independently. (3)
  Integrating multiple modalities: all single-modal encoders are aligned with the
  LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
  Our experiments show that X-LLM demonstrates impressive multimodel chat
  abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen
  images/instructions, and yields a 84.5\% relative score compared with GPT-4 on
  a synthetic multimodal instruction-following dataset. And we also conduct
  quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote
  the era of LLM-based speech recognition.
paper_authors._id: 64599eba4fe72fae522b3322
paper_authors.name: Feilong Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-02T14:19:38.048Z
paper_authors._id: 64127917227028b63e31fec8
paper_authors.avatarUrl: /avatars/23d7b088a5c8bdf8fb2e173216ffe316.svg
paper_authors.fullname: Feilong Chen
paper_authors.type: user
paper_authors.user: phellonchen
paper_authors._id: 64599eba4fe72fae522b3323
paper_authors.name: Minglun Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-09T08:52:33.300Z
paper_authors._id: 62fa3fdb0697d224219b38fa
paper_authors.avatarUrl: /avatars/38297ed87c87e1dd58feedaffc355218.svg
paper_authors.fullname: mlhan
paper_authors.type: user
paper_authors.user: mlhan
paper_authors._id: 64599eba4fe72fae522b3324
paper_authors.name: Haozhi Zhao
paper_authors._id: 64599eba4fe72fae522b3325
paper_authors.name: Qingyang Zhang
paper_authors._id: 64599eba4fe72fae522b3326
paper_authors.name: Jing Shi
paper_authors._id: 64599eba4fe72fae522b3327
paper_authors.name: Shuang Xu
paper_authors._id: 64599eba4fe72fae522b3328
paper_authors.name: Bo Xu
paper_title: X-LLM: Bootstrapping Advanced Large Language Models by Treating
    Multi-Modalities as Foreign Languages
paper_summary: Large language models (LLMs) have demonstrated remarkable language abilities.
  GPT-4, based on advanced LLMs, exhibits extraordinary multimodal capabilities
  beyond previous visual language models. We attribute this to the use of more
  advanced LLMs compared with previous multimodal models. Unfortunately, the
  model architecture and training strategies of GPT-4 are unknown. To endow LLMs
  with multimodal capabilities, we propose X-LLM, which converts Multi-modalities
  (images, speech, videos) into foreign languages using X2L interfaces and inputs
  them into a large Language model (ChatGLM). Specifically, X-LLM aligns multiple
  frozen single-modal encoders and a frozen LLM using X2L interfaces, where ``X''
  denotes multi-modalities such as image, speech, and videos, and ``L'' denotes
  languages. X-LLM's training consists of three stages: (1) Converting Multimodal
  Information: The first stage trains each X2L interface to align with its
  respective single-modal encoder separately to convert multimodal information
  into languages. (2) Aligning X2L representations with the LLM: single-modal
  encoders are aligned with the LLM through X2L interfaces independently. (3)
  Integrating multiple modalities: all single-modal encoders are aligned with the
  LLM through X2L interfaces to integrate multimodal capabilities into the LLM.
  Our experiments show that X-LLM demonstrates impressive multimodel chat
  abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen
  images/instructions, and yields a 84.5\% relative score compared with GPT-4 on
  a synthetic multimodal instruction-following dataset. And we also conduct
  quantitative tests on using LLM for ASR and multimodal ASR, hoping to promote
  the era of LLM-based speech recognition.
paper_ai_keywords: large language models (LLMs) multimodal capabilities GPT-4 visual language models model architecture training strategies X-LLM Multi-modalities X2L interfaces ChatGLM single-modal encoders multimodal information aligning X2L representations integrating multiple modalities multimodel chat abilities synthetic multimodal instruction-following dataset ASR multimodal ASR LLM-based speech recognition