title: LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
summary: Recent advancements in speech-to-speech dialogue systems leverage LLMs for
  multimodal interactions, yet they remain hindered by fine-tuning requirements,
  high computational overhead, and text-speech misalignment. Existing
  speech-enabled LLMs often degrade conversational quality by modifying the LLM,
  thereby compromising its linguistic capabilities. In contrast, we propose
  LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS
  system that generates high-quality speech with low latency, while fully
  preserving the capabilities of the base LLM. Our approach achieves a
  significantly lower Word Error Rate compared to speech-enabled LLMs, while
  operating at comparable latency and UTMOS score. By decoupling speech synthesis
  from LLM processing via a multi-queue token streaming system, LLMVoX supports
  seamless, infinite-length dialogues. Its plug-and-play design also facilitates
  extension to various tasks with different backbones. Furthermore, LLMVoX
  generalizes to new languages with only dataset adaptation, attaining a low
  Character Error Rate on an Arabic speech task. Additionally, we have integrated
  LLMVoX with a Vision-Language Model to create an omni-model with speech, text,
  and vision capabilities, without requiring additional multimodal training. Our
  code base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .
paper_authors._id: 67cacfd85dc0bf8f5e6fc803
paper_authors.name: Sambal Shikhar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:02:20.166Z
paper_authors._id: 6698db46157ceb76c49a823b
paper_authors.avatarUrl: /avatars/69a5a3fcd7f8aa66ae4d16d779afc5fe.svg
paper_authors.fullname: shikhar
paper_authors.type: user
paper_authors.user: sambal
paper_authors._id: 67cacfd85dc0bf8f5e6fc804
paper_authors.name: Mohammed Irfan Kurpath
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:37:55.973Z
paper_authors._id: 650289dbc130d99814b34dc5
paper_authors.avatarUrl: /avatars/ff0cf5add144cd79c41a255f41f34efb.svg
paper_authors.fullname: K Mohammed Irfan
paper_authors.type: user
paper_authors.user: k-m-irfan
paper_authors._id: 67cacfd85dc0bf8f5e6fc805
paper_authors.name: Sahal Shaji Mullappilly
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-07T13:37:53.972Z
paper_authors._id: 62e23c7f555a866437a53cd0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62e23c7f555a866437a53cd0/UaAsYZQXuwb4NSG5WnvdG.jpeg
paper_authors.fullname: Sahal Shaji
paper_authors.type: user
paper_authors.user: sahalshajim
paper_authors._id: 67cacfd85dc0bf8f5e6fc806
paper_authors.name: Jean Lahoud
paper_authors._id: 67cacfd85dc0bf8f5e6fc807
paper_authors.name: Fahad Khan
paper_authors._id: 67cacfd85dc0bf8f5e6fc808
paper_authors.name: Rao Muhammad Anwer
paper_authors._id: 67cacfd85dc0bf8f5e6fc809
paper_authors.name: Salman Khan
paper_authors._id: 67cacfd85dc0bf8f5e6fc80a
paper_authors.name: Hisham Cholakkal
paper_title: LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
paper_summary: Recent advancements in speech-to-speech dialogue systems leverage LLMs for
  multimodal interactions, yet they remain hindered by fine-tuning requirements,
  high computational overhead, and text-speech misalignment. Existing
  speech-enabled LLMs often degrade conversational quality by modifying the LLM,
  thereby compromising its linguistic capabilities. In contrast, we propose
  LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS
  system that generates high-quality speech with low latency, while fully
  preserving the capabilities of the base LLM. Our approach achieves a
  significantly lower Word Error Rate compared to speech-enabled LLMs, while
  operating at comparable latency and UTMOS score. By decoupling speech synthesis
  from LLM processing via a multi-queue token streaming system, LLMVoX supports
  seamless, infinite-length dialogues. Its plug-and-play design also facilitates
  extension to various tasks with different backbones. Furthermore, LLMVoX
  generalizes to new languages with only dataset adaptation, attaining a low
  Character Error Rate on an Arabic speech task. Additionally, we have integrated
  LLMVoX with a Vision-Language Model to create an omni-model with speech, text,
  and vision capabilities, without requiring additional multimodal training. Our
  code base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .
paper_ai_keywords: LLMs (Large Language Models) multimodal interactions fine-tuning computational overhead text-speech misalignment autoregressive streaming TTS (Text-to-Speech) low latency Word Error Rate UTMOS score multi-queue token streaming seamless, infinite-length dialogues plug-and-play design dataset adaptation Character Error Rate Vision-Language Model omni-model