title: LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive
    Modeling of Audio Discrete Codes
summary: Prior works have demonstrated zero-shot text-to-speech by using a generative
  language model on audio tokens obtained via a neural audio codec. It is still
  challenging, however, to adapt them to low-latency scenarios. In this paper, we
  present LiveSpeech - a fully autoregressive language model-based approach for
  zero-shot text-to-speech, enabling low-latency streaming of the output audio.
  To allow multiple token prediction within a single decoding step, we propose
  (1) using adaptive codebook loss weights that consider codebook contribution in
  each frame and focus on hard instances, and (2) grouping codebooks and
  processing groups in parallel. Experiments show our proposed models achieve
  competitive results to state-of-the-art baselines in terms of content accuracy,
  speaker similarity, audio quality, and inference speed while being suitable for
  low-latency streaming applications.
paper_authors._id: 6661279f3902e37d2d90bc5c
paper_authors.name: Trung Dang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-06T15:01:05.958Z
paper_authors._id: 66046dc99fcaed55efc28bb2
paper_authors.avatarUrl: /avatars/673bd6d1a5c6d58073b5feeea9c7c5e1.svg
paper_authors.fullname: Trung Dang
paper_authors.type: user
paper_authors.user: trungvd
paper_authors._id: 6661279f3902e37d2d90bc5d
paper_authors.name: David Aponte
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-06T08:46:25.079Z
paper_authors._id: 65b8003232a393895aff581d
paper_authors.avatarUrl: /avatars/19dc5be2c790b7be73301a0beb9df94a.svg
paper_authors.fullname: David Aponte
paper_authors.type: user
paper_authors.user: bambamcito
paper_authors._id: 6661279f3902e37d2d90bc5e
paper_authors.name: Dung Tran
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-09T20:05:31.467Z
paper_authors._id: 6662197d69b8bbc79f5815b2
paper_authors.avatarUrl: /avatars/7025332adbf6b6c0a9e9c92979f69584.svg
paper_authors.fullname: Dung Tran
paper_authors.type: user
paper_authors.user: dungntran
paper_authors._id: 6661279f3902e37d2d90bc5f
paper_authors.name: Kazuhito Koishida
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-06-06T03:06:08.399Z
paper_authors._id: 633f6bb8ac3954492880415f
paper_authors.avatarUrl: /avatars/f3dd763d8e092f68cbc33469f148b337.svg
paper_authors.fullname: Kazuhito Koishida
paper_authors.type: user
paper_authors.user: koishida
paper_title: LiveSpeech: Low-Latency Zero-shot Text-to-Speech via Autoregressive
    Modeling of Audio Discrete Codes
paper_summary: Prior works have demonstrated zero-shot text-to-speech by using a generative
  language model on audio tokens obtained via a neural audio codec. It is still
  challenging, however, to adapt them to low-latency scenarios. In this paper, we
  present LiveSpeech - a fully autoregressive language model-based approach for
  zero-shot text-to-speech, enabling low-latency streaming of the output audio.
  To allow multiple token prediction within a single decoding step, we propose
  (1) using adaptive codebook loss weights that consider codebook contribution in
  each frame and focus on hard instances, and (2) grouping codebooks and
  processing groups in parallel. Experiments show our proposed models achieve
  competitive results to state-of-the-art baselines in terms of content accuracy,
  speaker similarity, audio quality, and inference speed while being suitable for
  low-latency streaming applications.
paper_ai_summary: A novel autoregressive language model approach, LiveSpeech, enables low-latency text-to-speech by optimizing token prediction and parallel codebook processing.
paper_ai_keywords: generative language model audio tokens neural audio codec adaptive codebook loss weights autoregressive language model low-latency text-to-speech codebooks parallel processing content accuracy speaker similarity audio quality inference speed