title: VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient
    Large Speech-Language Model
summary: With the growing requirement for natural human-computer interaction,
  speech-based systems receive increasing attention as speech is one of the most
  common forms of daily communication. However, the existing speech models still
  experience high latency when generating the first audio token during streaming,
  which poses a significant bottleneck for deployment. To address this issue, we
  propose VITA-Audio, an end-to-end large speech model with fast audio-text token
  generation. Specifically, we introduce a lightweight Multiple Cross-modal Token
  Prediction (MCTP) module that efficiently generates multiple audio tokens
  within a single model forward pass, which not only accelerates the inference
  but also significantly reduces the latency for generating the first audio in
  streaming scenarios. In addition, a four-stage progressive training strategy is
  explored to achieve model acceleration with minimal loss of speech quality. To
  our knowledge, VITA-Audio is the first multi-modal large language model capable
  of generating audio output during the first forward pass, enabling real-time
  conversational capabilities with minimal latency. VITA-Audio is fully
  reproducible and is trained on open-source data only. Experimental results
  demonstrate that our model achieves an inference speedup of 3~5x at the 7B
  parameter scale, but also significantly outperforms open-source models of
  similar model size on multiple benchmarks for automatic speech recognition
  (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.
paper_authors._id: 681b232d7167935b1e979e64
paper_authors.name: Zuwei Long
paper_authors._id: 681b232d7167935b1e979e65
paper_authors.name: Yunhang Shen
paper_authors._id: 681b232d7167935b1e979e66
paper_authors.name: Chaoyou Fu
paper_authors._id: 681b232d7167935b1e979e67
paper_authors.name: Heting Gao
paper_authors._id: 681b232d7167935b1e979e68
paper_authors.name: Lijiang Li
paper_authors._id: 681b232d7167935b1e979e69
paper_authors.name: Peixian Chen
paper_authors._id: 681b232d7167935b1e979e6a
paper_authors.name: Mengdan Zhang
paper_authors._id: 681b232d7167935b1e979e6b
paper_authors.name: Hang Shao
paper_authors._id: 681b232d7167935b1e979e6c
paper_authors.name: Jian Li
paper_authors._id: 681b232d7167935b1e979e6d
paper_authors.name: Jinlong Peng
paper_authors._id: 681b232d7167935b1e979e6e
paper_authors.name: Haoyu Cao
paper_authors._id: 681b232d7167935b1e979e6f
paper_authors.name: Ke Li
paper_authors._id: 681b232d7167935b1e979e70
paper_authors.name: Rongrong Ji
paper_authors._id: 681b232d7167935b1e979e71
paper_authors.name: Xing Sun
paper_title: VITA-Audio: Fast Interleaved Cross-Modal Token Generation for Efficient
    Large Speech-Language Model
paper_summary: With the growing requirement for natural human-computer interaction,
  speech-based systems receive increasing attention as speech is one of the most
  common forms of daily communication. However, the existing speech models still
  experience high latency when generating the first audio token during streaming,
  which poses a significant bottleneck for deployment. To address this issue, we
  propose VITA-Audio, an end-to-end large speech model with fast audio-text token
  generation. Specifically, we introduce a lightweight Multiple Cross-modal Token
  Prediction (MCTP) module that efficiently generates multiple audio tokens
  within a single model forward pass, which not only accelerates the inference
  but also significantly reduces the latency for generating the first audio in
  streaming scenarios. In addition, a four-stage progressive training strategy is
  explored to achieve model acceleration with minimal loss of speech quality. To
  our knowledge, VITA-Audio is the first multi-modal large language model capable
  of generating audio output during the first forward pass, enabling real-time
  conversational capabilities with minimal latency. VITA-Audio is fully
  reproducible and is trained on open-source data only. Experimental results
  demonstrate that our model achieves an inference speedup of 3~5x at the 7B
  parameter scale, but also significantly outperforms open-source models of
  similar model size on multiple benchmarks for automatic speech recognition
  (ASR), text-to-speech (TTS), and spoken question answering (SQA) tasks.
paper_ai_summary: VITA-Audio, an end-to-end speech model, introduces a lightweight MCTP module to generate multiple audio tokens efficiently, reducing latency in streaming applications with enhanced speech processing capabilities.
paper_ai_keywords: Multiple Cross-modal Token Prediction (MCTP) audio tokens inference speedup automatic speech recognition (ASR) text-to-speech (TTS) spoken question answering (SQA) progressive training strategy