SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal
  Conversational Abilities
Multi-modal large language models are regarded as a crucial step towards
Artificial General Intelligence (AGI) and have garnered significant interest
with the emergence of ChatGPT. However, current speech-language models
typically adopt the cascade paradigm, preventing inter-modal knowledge
transfer. In this paper, we propose SpeechGPT, a large language model with
intrinsic cross-modal conversational abilities, capable of perceiving and
generating multi-model content. With discrete speech representations, we first
construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.
Additionally, we employ a three-stage training strategy that includes
modality-adaptation pre-training, cross-modal instruction fine-tuning, and
chain-of-modality instruction fine-tuning. The experimental results demonstrate
that SpeechGPT has an impressive capacity to follow multi-modal human
instructions and highlight the potential of handling multiple modalities with
one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
646704a856372fc9e3e99f7c
Dong Zhang
claimed_verified
2024-03-02T19:14:55.077Z
6377596f82ce7c3b8a528f72
/avatars/f55676883e054d7e4a3b96ba1e752a80.svg
Dong Zhang
user
nutation
646704a856372fc9e3e99f7d
Shimin Li
646704a856372fc9e3e99f7e
Xin Zhang
claimed_verified
2024-02-21T08:31:52.952Z
64eda039067fbb625f851207
/avatars/fb6014110c32ab0d46a63558d448e00d.svg
zhangxin
user
xin98
646704a856372fc9e3e99f7f
Jun Zhan
646704a856372fc9e3e99f80
Pengyu Wang
646704a856372fc9e3e99f81
Yaqian Zhou
646704a856372fc9e3e99f82
Xipeng Qiu
admin_assigned
2023-05-21T20:38:05.614Z
61457b8deff2c9fdb4de4988
https://cdn-avatars.huggingface.co/v1/production/uploads/1632381702899-61457b8deff2c9fdb4de4988.jpeg
Xipeng Qiu
user
xpqiu
SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal
  Conversational Abilities
Multi-modal large language models are regarded as a crucial step towards
Artificial General Intelligence (AGI) and have garnered significant interest
with the emergence of ChatGPT. However, current speech-language models
typically adopt the cascade paradigm, preventing inter-modal knowledge
transfer. In this paper, we propose SpeechGPT, a large language model with
intrinsic cross-modal conversational abilities, capable of perceiving and
generating multi-model content. With discrete speech representations, we first
construct SpeechInstruct, a large-scale cross-modal speech instruction dataset.
Additionally, we employ a three-stage training strategy that includes
modality-adaptation pre-training, cross-modal instruction fine-tuning, and
chain-of-modality instruction fine-tuning. The experimental results demonstrate
that SpeechGPT has an impressive capacity to follow multi-modal human
instructions and highlight the potential of handling multiple modalities with
one model. Demos are shown in https://0nutation.github.io/SpeechGPT.github.io/.
SpeechGPT, a large language model with intrinsic cross-modal conversational abilities, demonstrates impressive performance in following multi-modal human instructions through a three-stage training strategy.
multi-modal large language models AGI SpeechGPT cross-modal conversational abilities discrete speech representations SpeechInstruct modality-adaptation pre-training cross-modal instruction fine-tuning chain-of-modality instruction fine-tuning