VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient
  Large Speech-Language Model
With the growing requirement for natural human-computer interaction,
speech-based systems receive increasing attention as speech is one of the most
common forms of daily communication. However, the existing speech models still
experience high latency when generating the first audio token during streaming,
which poses a significant bottleneck for deployment. To address this issue, we
propose VITA-Audio, an end-to-end large speech model with fast audio-text token
generation. Specifically, we introduce a lightweight Multiple Cross-modal Token
Prediction (MCTP) module that efficiently generates multiple audio tokens
within a single model forward pass, which not only accelerates the inference
but also significantly reduces the latency for generating the first audio in
streaming scenarios. In addition, a four-stage progressive training strategy is
explored to achieve model acceleration with minimal loss of speech quality. To
our knowledge, VITA-Audio is the first multi-modal large language model capable
of generating audio output during the first forward pass, enabling real-time
conversational capabilities with minimal latency. VITA-Audio is fully
reproducible and is trained on open-source data only. Experimental results
demonstrate that our model achieves an inference speedup of 3~5x at the 7B
parameter scale, but also significantly outperforms open-source models of
similar model size on multiple benchmarks for automatic speech recognition
(ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.
681b232d7167935b1e979e64
Zuwei Long
681b232d7167935b1e979e65
Yunhang Shen
681b232d7167935b1e979e66
Chaoyou Fu
681b232d7167935b1e979e67
Heting Gao
681b232d7167935b1e979e68
Lijiang Li
681b232d7167935b1e979e69
Peixian Chen
681b232d7167935b1e979e6a
Mengdan Zhang
681b232d7167935b1e979e6b
Hang Shao
681b232d7167935b1e979e6c
Jian Li
681b232d7167935b1e979e6d
Jinlong Peng
681b232d7167935b1e979e6e
Haoyu Cao
681b232d7167935b1e979e6f
Ke Li
681b232d7167935b1e979e70
Rongrong Ji
681b232d7167935b1e979e71
Xing Sun
VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient
  Large Speech-Language Model
With the growing requirement for natural human-computer interaction,
speech-based systems receive increasing attention as speech is one of the most
common forms of daily communication. However, the existing speech models still
experience high latency when generating the first audio token during streaming,
which poses a significant bottleneck for deployment. To address this issue, we
propose VITA-Audio, an end-to-end large speech model with fast audio-text token
generation. Specifically, we introduce a lightweight Multiple Cross-modal Token
Prediction (MCTP) module that efficiently generates multiple audio tokens
within a single model forward pass, which not only accelerates the inference
but also significantly reduces the latency for generating the first audio in
streaming scenarios. In addition, a four-stage progressive training strategy is
explored to achieve model acceleration with minimal loss of speech quality. To
our knowledge, VITA-Audio is the first multi-modal large language model capable
of generating audio output during the first forward pass, enabling real-time
conversational capabilities with minimal latency. VITA-Audio is fully
reproducible and is trained on open-source data only. Experimental results
demonstrate that our model achieves an inference speedup of 3~5x at the 7B
parameter scale, but also significantly outperforms open-source models of
similar model size on multiple benchmarks for automatic speech recognition
(ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.
VITA-Audio, an end-to-end speech model, introduces a lightweight MCTP module to generate multiple audio tokens efficiently, reducing latency in streaming applications with enhanced speech processing capabilities.
Multiple Cross-modal Token Prediction (MCTP) audio tokens inference speedup automatic speech recognition (ASR) text-to-speech (TTS) spoken question answering (SQA) progressive training strategy