title: NExT-GPT: Any-to-Any Multimodal LLM
summary: While recently Multimodal Large Language Models (MM-LLMs) have made exciting
  strides, they mostly fall prey to the limitation of only input-side multimodal
  understanding, without the ability to produce content in multiple modalities.
  As we humans always perceive the world and communicate with people through
  various modalities, developing any-to-any MM-LLMs capable of accepting and
  delivering content in any modality becomes essential to human-level AI. To fill
  the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
  NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
  decoders, enabling NExT-GPT to perceive inputs and generate outputs in
  arbitrary combinations of text, images, videos, and audio. By leveraging the
  existing well-trained highly-performing encoders and decoders, NExT-GPT is
  tuned with only a small amount of parameter (1%) of certain projection layers,
  which not only benefits low-cost training and also facilitates convenient
  expansion to more potential modalities. Moreover, we introduce a
  modality-switching instruction tuning (MosIT) and manually curate a
  high-quality dataset for MosIT, based on which NExT-GPT is empowered with
  complex cross-modal semantic understanding and content generation. Overall, our
  research showcases the promising possibility of building an AI agent capable of
  modeling universal modalities, paving the way for more human-like AI research
  in the community.
paper_authors._id: 64ffd83052aec889f49b61ea
paper_authors.name: Shengqiong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:00:46.264Z
paper_authors._id: 64c139d867eff857ea51caa8
paper_authors.avatarUrl: /avatars/4b7b3f41c2e2cfa21dd43bbac6e081ae.svg
paper_authors.fullname: Shengqiong Wu
paper_authors.type: user
paper_authors.user: ChocoWu
paper_authors._id: 64ffd83052aec889f49b61eb
paper_authors.name: Hao Fei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:01:39.654Z
paper_authors._id: 647773a1168cb428e00e9a8f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647773a1168cb428e00e9a8f/NiRR3ScY6Plzjibfwy1hC.jpeg
paper_authors.fullname: Hao Fei
paper_authors.type: user
paper_authors.user: scofield7419
paper_authors._id: 64ffd83052aec889f49b61ec
paper_authors.name: Leigang Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-12T11:01:47.187Z
paper_authors._id: 641a6e324182690729c860b5
paper_authors.avatarUrl: /avatars/4599451d9775bf1aa11ef1e7011cb05a.svg
paper_authors.fullname: Leigang Qu
paper_authors.type: user
paper_authors.user: leigangqu
paper_authors._id: 64ffd83052aec889f49b61ed
paper_authors.name: Wei Ji
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-14T07:31:49.655Z
paper_authors._id: 65025db534ddd20327363af6
paper_authors.avatarUrl: /avatars/ef7982780d2bc544882ed32441c0df5d.svg
paper_authors.fullname: Wei Ji
paper_authors.type: user
paper_authors.user: away0523
paper_authors._id: 64ffd83052aec889f49b61ee
paper_authors.name: Tat-Seng Chua
paper_title: NExT-GPT: Any-to-Any Multimodal LLM
paper_summary: While recently Multimodal Large Language Models (MM-LLMs) have made exciting
  strides, they mostly fall prey to the limitation of only input-side multimodal
  understanding, without the ability to produce content in multiple modalities.
  As we humans always perceive the world and communicate with people through
  various modalities, developing any-to-any MM-LLMs capable of accepting and
  delivering content in any modality becomes essential to human-level AI. To fill
  the gap, we present an end-to-end general-purpose any-to-any MM-LLM system,
  NExT-GPT. We connect an LLM with multimodal adaptors and different diffusion
  decoders, enabling NExT-GPT to perceive inputs and generate outputs in
  arbitrary combinations of text, images, videos, and audio. By leveraging the
  existing well-trained highly-performing encoders and decoders, NExT-GPT is
  tuned with only a small amount of parameter (1%) of certain projection layers,
  which not only benefits low-cost training and also facilitates convenient
  expansion to more potential modalities. Moreover, we introduce a
  modality-switching instruction tuning (MosIT) and manually curate a
  high-quality dataset for MosIT, based on which NExT-GPT is empowered with
  complex cross-modal semantic understanding and content generation. Overall, our
  research showcases the promising possibility of building an AI agent capable of
  modeling universal modalities, paving the way for more human-like AI research
  in the community.
paper_ai_keywords: Multimodal Large Language Models (MM-LLMs) input-side multimodal understanding CNNs diffusion decoders LLM multimodal adaptors any-to-any MM-LLMs text images videos audio parameter-efficient fine-tuning well-trained encoders decoders modality-switching instruction (MosIT) complex cross-modal semantic understanding cross-modal content generation