title: Voila: Voice-Language Foundation Models for Real-Time Autonomous
    Interaction and Voice Role-Play
summary: A voice AI agent that blends seamlessly into daily life would interact with
  humans in an autonomous, real-time, and emotionally expressive manner. Rather
  than merely reacting to commands, it would continuously listen, reason, and
  respond proactively, fostering fluid, dynamic, and emotionally resonant
  interactions. We introduce Voila, a family of large voice-language foundation
  models that make a step towards this vision. Voila moves beyond traditional
  pipeline systems by adopting a new end-to-end architecture that enables
  full-duplex, low-latency conversations while preserving rich vocal nuances such
  as tone, rhythm, and emotion. It achieves a response latency of just 195
  milliseconds, surpassing the average human response time. Its hierarchical
  multi-scale Transformer integrates the reasoning capabilities of large language
  models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware
  voice generation -- where users can simply write text instructions to define
  the speaker's identity, tone, and other characteristics. Moreover, Voila
  supports over one million pre-built voices and efficient customization of new
  ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue,
  Voila is designed as a unified model for a wide range of voice-based
  applications, including automatic speech recognition (ASR), Text-to-Speech
  (TTS), and, with minimal adaptation, multilingual speech translation. Voila is
  fully open-sourced to support open research and accelerate progress toward
  next-generation human-machine interactions.
paper_authors._id: 6819982f17007d963b9d4166
paper_authors.name: Yemin Shi
paper_authors._id: 6819982f17007d963b9d4167
paper_authors.name: Yu Shu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-12T18:34:56.108Z
paper_authors._id: 670f584caddfb18348a52240
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/rJCWRGII2-Kn6nc1-v44J.jpeg
paper_authors.fullname: YuShu
paper_authors.type: user
paper_authors.user: TabbbSY
paper_authors._id: 6819982f17007d963b9d4168
paper_authors.name: Siwei Dong
paper_authors._id: 6819982f17007d963b9d4169
paper_authors.name: Guangyi Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:46:52.667Z
paper_authors._id: 6108ae87823007eaf0c7bd1e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6108ae87823007eaf0c7bd1e/dKjdx9I5waJs6oUQ0_mmT.png
paper_authors.fullname: Guangyi Liu
paper_authors.type: user
paper_authors.user: guangyil
paper_authors._id: 6819982f17007d963b9d416a
paper_authors.name: Jaward Sesay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-06T08:32:48.746Z
paper_authors._id: 6438a9027de34e8ea7e4b257
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6438a9027de34e8ea7e4b257/vib8QSd1AWMr_bR9ig_xJ.jpeg
paper_authors.fullname: Jaward Sesay
paper_authors.type: user
paper_authors.user: Jaward
paper_authors._id: 6819982f17007d963b9d416b
paper_authors.name: Jingwen Li
paper_authors._id: 6819982f17007d963b9d416c
paper_authors.name: Zhiting Hu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T08:46:15.191Z
paper_authors._id: 665bfa1b0d71762b8613282d
paper_authors.avatarUrl: /avatars/edbde7b1b47032339a1ecc59f8ea8f1a.svg
paper_authors.fullname: Zhiting Hu
paper_authors.type: user
paper_authors.user: zhitinghu
paper_title: Voila: Voice-Language Foundation Models for Real-Time Autonomous
    Interaction and Voice Role-Play
paper_summary: A voice AI agent that blends seamlessly into daily life would interact with
  humans in an autonomous, real-time, and emotionally expressive manner. Rather
  than merely reacting to commands, it would continuously listen, reason, and
  respond proactively, fostering fluid, dynamic, and emotionally resonant
  interactions. We introduce Voila, a family of large voice-language foundation
  models that make a step towards this vision. Voila moves beyond traditional
  pipeline systems by adopting a new end-to-end architecture that enables
  full-duplex, low-latency conversations while preserving rich vocal nuances such
  as tone, rhythm, and emotion. It achieves a response latency of just 195
  milliseconds, surpassing the average human response time. Its hierarchical
  multi-scale Transformer integrates the reasoning capabilities of large language
  models (LLMs) with powerful acoustic modeling, enabling natural, persona-aware
  voice generation -- where users can simply write text instructions to define
  the speaker's identity, tone, and other characteristics. Moreover, Voila
  supports over one million pre-built voices and efficient customization of new
  ones from brief audio samples as short as 10 seconds. Beyond spoken dialogue,
  Voila is designed as a unified model for a wide range of voice-based
  applications, including automatic speech recognition (ASR), Text-to-Speech
  (TTS), and, with minimal adaptation, multilingual speech translation. Voila is
  fully open-sourced to support open research and accelerate progress toward
  next-generation human-machine interactions.
paper_ai_summary: Voila is an end-to-end voice-language model that enables low-latency, emotionally expressive voice interactions and supports multiple applications including ASR, TTS, and multilingual speech translation.
paper_ai_keywords: end-to-end architecture full-duplex low-latency conversations hierarchical multi-scale Transformer acoustic modeling voice generation persona-aware automatic speech recognition (ASR) Text-to-Speech (TTS) multilingual speech translation