LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
Recent advancements in speech-to-speech dialogue systems leverage LLMs for
multimodal interactions, yet they remain hindered by fine-tuning requirements,
high computational overhead, and text-speech misalignment. Existing
speech-enabled LLMs often degrade conversational quality by modifying the LLM,
thereby compromising its linguistic capabilities. In contrast, we propose
LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS
system that generates high-quality speech with low latency, while fully
preserving the capabilities of the base LLM. Our approach achieves a
significantly lower Word Error Rate compared to speech-enabled LLMs, while
operating at comparable latency and UTMOS score. By decoupling speech synthesis
from LLM processing via a multi-queue token streaming system, LLMVoX supports
seamless, infinite-length dialogues. Its plug-and-play design also facilitates
extension to various tasks with different backbones. Furthermore, LLMVoX
generalizes to new languages with only dataset adaptation, attaining a low
Character Error Rate on an Arabic speech task. Additionally, we have integrated
LLMVoX with a Vision-Language Model to create an omni-model with speech, text,
and vision capabilities, without requiring additional multimodal training. Our
code base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .
67cacfd85dc0bf8f5e6fc803
Sambal Shikhar
claimed_verified
2025-03-10T08:02:20.166Z
6698db46157ceb76c49a823b
/avatars/69a5a3fcd7f8aa66ae4d16d779afc5fe.svg
shikhar
user
sambal
67cacfd85dc0bf8f5e6fc804
Mohammed Irfan Kurpath
claimed_verified
2025-03-07T13:37:55.973Z
650289dbc130d99814b34dc5
/avatars/ff0cf5add144cd79c41a255f41f34efb.svg
K Mohammed Irfan
user
k-m-irfan
67cacfd85dc0bf8f5e6fc805
Sahal Shaji Mullappilly
claimed_verified
2025-03-07T13:37:53.972Z
62e23c7f555a866437a53cd0
https://cdn-avatars.huggingface.co/v1/production/uploads/62e23c7f555a866437a53cd0/UaAsYZQXuwb4NSG5WnvdG.jpeg
Sahal Shaji
user
sahalshajim
67cacfd85dc0bf8f5e6fc806
Jean Lahoud
67cacfd85dc0bf8f5e6fc807
Fahad Khan
67cacfd85dc0bf8f5e6fc808
Rao Muhammad Anwer
67cacfd85dc0bf8f5e6fc809
Salman Khan
67cacfd85dc0bf8f5e6fc80a
Hisham Cholakkal
LLMVoX: Autoregressive Streaming Text-to-Speech Model for Any LLM
Recent advancements in speech-to-speech dialogue systems leverage LLMs for
multimodal interactions, yet they remain hindered by fine-tuning requirements,
high computational overhead, and text-speech misalignment. Existing
speech-enabled LLMs often degrade conversational quality by modifying the LLM,
thereby compromising its linguistic capabilities. In contrast, we propose
LLMVoX, a lightweight 30M-parameter, LLM-agnostic, autoregressive streaming TTS
system that generates high-quality speech with low latency, while fully
preserving the capabilities of the base LLM. Our approach achieves a
significantly lower Word Error Rate compared to speech-enabled LLMs, while
operating at comparable latency and UTMOS score. By decoupling speech synthesis
from LLM processing via a multi-queue token streaming system, LLMVoX supports
seamless, infinite-length dialogues. Its plug-and-play design also facilitates
extension to various tasks with different backbones. Furthermore, LLMVoX
generalizes to new languages with only dataset adaptation, attaining a low
Character Error Rate on an Arabic speech task. Additionally, we have integrated
LLMVoX with a Vision-Language Model to create an omni-model with speech, text,
and vision capabilities, without requiring additional multimodal training. Our
code base and project page is available at https://mbzuai-oryx.github.io/LLMVoX .
LLMVoX is a lightweight, LLM-agnostic TTS system that decouples speech synthesis from model processing, ensuring high-quality speech with low latency and supporting seamless multimodal dialogue.
LLM multimodal interactions fine-tuning computational overhead text-speech misalignment autoregressive streaming TTS Word Error Rate latency UTMOS score multi-queue token streaming infinite-length dialogues plug-and-play dataset adaptation Character Error Rate Vision-Language Model omni-model