title: SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal
    Conversational Abilities
summary: Multi-modal large language models are regarded as a crucial step towards
  Artificial General Intelligence (AGI) and have garnered significant interest
  with the emergence of ChatGPT. However, current speech-language models
  typically adopt the cascade paradigm, preventing inter-modal knowledge
  transfer. In this paper, we propose SpeechGPT, a large language model with
  intrinsic cross-modal conversational abilities, capable of perceiving and
  generating multi-model content. With discrete speech representations, we first
  construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.
  Additionally, we employ a three-stage training strategy that includes
  modality-adaptation pre-training, cross-modal instruction fine-tuning, and
  chain-of-modality instruction fine-tuning. The experimental results demonstrate
  that SpeechGPT has an impressive capacity to follow multi-modal human
  instructions and highlight the potential of handling multiple modalities with
  one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
paper_authors._id: 646704a856372fc9e3e99f7c
paper_authors.name: Dong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-02T19:14:55.077Z
paper_authors._id: 6377596f82ce7c3b8a528f72
paper_authors.avatarUrl: /avatars/f55676883e054d7e4a3b96ba1e752a80.svg
paper_authors.fullname: Dong Zhang
paper_authors.type: user
paper_authors.user: nutation
paper_authors._id: 646704a856372fc9e3e99f7d
paper_authors.name: Shimin Li
paper_authors._id: 646704a856372fc9e3e99f7e
paper_authors.name: Xin Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-21T08:31:52.952Z
paper_authors._id: 64eda039067fbb625f851207
paper_authors.avatarUrl: /avatars/fb6014110c32ab0d46a63558d448e00d.svg
paper_authors.fullname: zhangxin
paper_authors.type: user
paper_authors.user: xin98
paper_authors._id: 646704a856372fc9e3e99f7f
paper_authors.name: Jun Zhan
paper_authors._id: 646704a856372fc9e3e99f80
paper_authors.name: Pengyu Wang
paper_authors._id: 646704a856372fc9e3e99f81
paper_authors.name: Yaqian Zhou
paper_authors._id: 646704a856372fc9e3e99f82
paper_authors.name: Xipeng Qiu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-21T20:38:05.614Z
paper_authors._id: 61457b8deff2c9fdb4de4988
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
paper_authors.fullname: Xipeng Qiu
paper_authors.type: user
paper_authors.user: xpqiu
paper_title: SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal
    Conversational Abilities
paper_summary: Multi-modal large language models are regarded as a crucial step towards
  Artificial General Intelligence (AGI) and have garnered significant interest
  with the emergence of ChatGPT. However, current speech-language models
  typically adopt the cascade paradigm, preventing inter-modal knowledge
  transfer. In this paper, we propose SpeechGPT, a large language model with
  intrinsic cross-modal conversational abilities, capable of perceiving and
  generating multi-model content. With discrete speech representations, we first
  construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.
  Additionally, we employ a three-stage training strategy that includes
  modality-adaptation pre-training, cross-modal instruction fine-tuning, and
  chain-of-modality instruction fine-tuning. The experimental results demonstrate
  that SpeechGPT has an impressive capacity to follow multi-modal human
  instructions and highlight the potential of handling multiple modalities with
  one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
paper_ai_keywords: multi-modal large language models Artificial General Intelligence (AGI) SpeechGPT intrinsic cross-modal conversational abilities SpeechInstruct large-scale cross-modal speech instruction dataset three-stage training strategy modality-adaptation pre-training cross-modal instruction fine-tuning chain-of-modality instructions fine-tuning multi-modal human instructions