title: AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
summary: We introduce AnyGPT, an any-to-any multimodal language model that utilizes
  discrete representations for the unified processing of various modalities,
  including speech, text, images, and music. AnyGPT can be trained stably without
  any alterations to the current large language model (LLM) architecture or
  training paradigms. Instead, it relies exclusively on data-level preprocessing,
  facilitating the seamless integration of new modalities into LLMs, akin to the
  incorporation of new languages. We build a multimodal text-centric dataset for
  multimodal alignment pre-training. Utilizing generative models, we synthesize
  the first large-scale any-to-any multimodal instruction dataset. It consists of
  108k samples of multi-turn conversations that intricately interweave various
  modalities, thus equipping the model to handle arbitrary combinations of
  multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is
  capable of facilitating any-to-any multimodal conversation while achieving
  performance comparable to specialized models across all modalities, proving
  that discrete representations can effectively and conveniently unify multiple
  modalities within a language model. Demos are shown in
  https://junzhan2000.github.io/AnyGPT.github.io/
paper_authors._id: 65d441b42f7b897bfaa26089
paper_authors.name: Jun Zhan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:34:18.222Z
paper_authors._id: 6509858fa2abcb18d633597b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6509858fa2abcb18d633597b/uxfPgCvhL1Tzw_lDR9hW-.jpeg
paper_authors.fullname: JunZhan
paper_authors.type: user
paper_authors.user: zhanjun
paper_authors._id: 65d441b42f7b897bfaa2608a
paper_authors.name: Junqi Dai
paper_authors._id: 65d441b42f7b897bfaa2608b
paper_authors.name: Jiasheng Ye
paper_authors._id: 65d441b42f7b897bfaa2608c
paper_authors.name: Yunhua Zhou
paper_authors._id: 65d441b42f7b897bfaa2608d
paper_authors.name: Dong Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:36:12.891Z
paper_authors._id: 6377596f82ce7c3b8a528f72
paper_authors.avatarUrl: /avatars/f55676883e054d7e4a3b96ba1e752a80.svg
paper_authors.fullname: Dong Zhang
paper_authors.type: user
paper_authors.user: nutation
paper_authors._id: 65d441b42f7b897bfaa2608e
paper_authors.name: Zhigeng Liu
paper_authors._id: 65d441b42f7b897bfaa2608f
paper_authors.name: Xin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:11.642Z
paper_authors._id: 64eda039067fbb625f851207
paper_authors.avatarUrl: /avatars/fb6014110c32ab0d46a63558d448e00d.svg
paper_authors.fullname: zhangxin
paper_authors.type: user
paper_authors.user: xin98
paper_authors._id: 65d441b42f7b897bfaa26090
paper_authors.name: Ruibin Yuan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:36:54.038Z
paper_authors._id: 5fd6f670053c8345eddc1b68
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5fd6f670053c8345eddc1b68/cuTsu2krRYHC6zYGD2dpQ.jpeg
paper_authors.fullname: Ruibin Yuan
paper_authors.type: user
paper_authors.user: a43992899
paper_authors._id: 65d441b42f7b897bfaa26091
paper_authors.name: Ge Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:37:43.902Z
paper_authors._id: 638efcf4c67af472d316d424
paper_authors.avatarUrl: /avatars/97a57859d7d87a3a8f1bb41d32a72bc2.svg
paper_authors.fullname: Ge Zhang
paper_authors.type: user
paper_authors.user: zhangysk
paper_authors._id: 65d441b42f7b897bfaa26092
paper_authors.name: Linyang Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-18T09:09:27.606Z
paper_authors._id: 6698ad87a9778d20ff373142
paper_authors.avatarUrl: /avatars/5e3546d124e8c4f4636a41eab6f0cf68.svg
paper_authors.fullname: linyangli
paper_authors.type: user
paper_authors.user: linyangli
paper_authors._id: 65d441b42f7b897bfaa26093
paper_authors.name: Hang Yan
paper_authors._id: 65d441b42f7b897bfaa26094
paper_authors.name: Jie Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:30:13.762Z
paper_authors._id: 641a6895fb5ffff5ac79d593
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/641a6895fb5ffff5ac79d593/vxvwsto3llOEWGqQKGMYx.jpeg
paper_authors.fullname: Jie Fu
paper_authors.type: user
paper_authors.user: bigaidream
paper_authors._id: 65d441b42f7b897bfaa26095
paper_authors.name: Tao Gui
paper_authors._id: 65d441b42f7b897bfaa26096
paper_authors.name: Tianxiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:39:28.011Z
paper_authors._id: 618c9e4cf35289d4bcab9d02
paper_authors.avatarUrl: /avatars/28b5b096190b539295ec0da167f1abf3.svg
paper_authors.fullname: Tianxiang Sun
paper_authors.type: user
paper_authors.user: txsun
paper_authors._id: 65d441b42f7b897bfaa26097
paper_authors.name: Yugang Jiang
paper_authors._id: 65d441b42f7b897bfaa26098
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-20T09:39:48.694Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_title: AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling
paper_summary: We introduce AnyGPT, an any-to-any multimodal language model that utilizes
  discrete representations for the unified processing of various modalities,
  including speech, text, images, and music. AnyGPT can be trained stably without
  any alterations to the current large language model (LLM) architecture or
  training paradigms. Instead, it relies exclusively on data-level preprocessing,
  facilitating the seamless integration of new modalities into LLMs, akin to the
  incorporation of new languages. We build a multimodal text-centric dataset for
  multimodal alignment pre-training. Utilizing generative models, we synthesize
  the first large-scale any-to-any multimodal instruction dataset. It consists of
  108k samples of multi-turn conversations that intricately interweave various
  modalities, thus equipping the model to handle arbitrary combinations of
  multimodal inputs and outputs. Experimental results demonstrate that AnyGPT is
  capable of facilitating any-to-any multimodal conversation while achieving
  performance comparable to specialized models across all modalities, proving
  that discrete representations can effectively and conveniently unify multiple
  modalities within a language model. Demos are shown in
  https://junzhan2000.github.io/AnyGPT.github.io/
paper_ai_summary: AnyGPT is a multimodal language model using discrete representations to process and generate content across various modalities with performance on par with specialized models.
paper_ai_keywords: discrete representations multimodal language model multimodal alignment pre-training generative models multimodal instruction dataset multi-turn conversations