title: Continuous Speech Synthesis using per-token Latent Diffusion
summary: The success of autoregressive transformer models with discrete tokens has
  inspired quantization-based approaches for continuous modalities, though these
  often limit reconstruction quality. We therefore introduce SALAD, a per-token
  latent diffusion model for zero-shot text-to-speech, that operates on
  continuous representations. SALAD builds upon the recently proposed expressive
  diffusion head for image generation, and extends it to generate variable-length
  outputs. Our approach utilizes semantic tokens for providing contextual
  information and determining the stopping condition. We suggest three continuous
  variants for our method, extending popular discrete speech synthesis
  techniques. Additionally, we implement discrete baselines for each variant and
  conduct a comparative analysis of discrete versus continuous speech modeling
  techniques. Our results demonstrate that both continuous and discrete
  approaches are highly competent, and that SALAD achieves a superior
  intelligibility score while obtaining speech quality and speaker similarity on
  par with the ground-truth audio.
paper_authors._id: 671753f39feb1b7d3a5f10c6
paper_authors.name: Arnon Turetzky
paper_authors._id: 671753f39feb1b7d3a5f10c7
paper_authors.name: Nimrod Shabtay
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T11:25:20.632Z
paper_authors._id: 62bedff7304b82a773bf8c1b
paper_authors.avatarUrl: /avatars/f9e79dc196caa95c220127c6212e9944.svg
paper_authors.fullname: Nimrod Shabtay
paper_authors.type: user
paper_authors.user: NimrodShabtay1986
paper_authors._id: 671753f39feb1b7d3a5f10c8
paper_authors.name: Slava Shechtman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:17:44.202Z
paper_authors._id: 66e9d97456fb59c436a1bbc1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/1y4m9Y9RZJ9lrw85pwELm.png
paper_authors.fullname: Slava Shechtman
paper_authors.type: user
paper_authors.user: slavashe
paper_authors._id: 671753f39feb1b7d3a5f10c9
paper_authors.name: Hagai Aronowitz
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:17:50.097Z
paper_authors._id: 643425b4a4c9c55871a7a02b
paper_authors.avatarUrl: /avatars/eb2f357888159f5120bbf70a40cb089d.svg
paper_authors.fullname: Hagai Aronowitz
paper_authors.type: user
paper_authors.user: hagaia
paper_authors._id: 671753f39feb1b7d3a5f10ca
paper_authors.name: David Haws
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:17:55.258Z
paper_authors._id: 66f59c2051064e3444ff38c1
paper_authors.avatarUrl: /avatars/b64fb3ab1910dc8746e243083a3ef549.svg
paper_authors.fullname: David Haws
paper_authors.type: user
paper_authors.user: dchaws
paper_authors._id: 671753f39feb1b7d3a5f10cb
paper_authors.name: Ron Hoory
paper_authors._id: 671753f39feb1b7d3a5f10cc
paper_authors.name: Avihu Dekel
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T07:55:44.222Z
paper_authors._id: 63b7e09359060ca9f4c4de35
paper_authors.avatarUrl: /avatars/3d8e6d46d9af81527e504dbd1324e4a1.svg
paper_authors.fullname: Avihu Dekel
paper_authors.type: user
paper_authors.user: Avihu
paper_title: Continuous Speech Synthesis using per-token Latent Diffusion
paper_summary: The success of autoregressive transformer models with discrete tokens has
  inspired quantization-based approaches for continuous modalities, though these
  often limit reconstruction quality. We therefore introduce SALAD, a per-token
  latent diffusion model for zero-shot text-to-speech, that operates on
  continuous representations. SALAD builds upon the recently proposed expressive
  diffusion head for image generation, and extends it to generate variable-length
  outputs. Our approach utilizes semantic tokens for providing contextual
  information and determining the stopping condition. We suggest three continuous
  variants for our method, extending popular discrete speech synthesis
  techniques. Additionally, we implement discrete baselines for each variant and
  conduct a comparative analysis of discrete versus continuous speech modeling
  techniques. Our results demonstrate that both continuous and discrete
  approaches are highly competent, and that SALAD achieves a superior
  intelligibility score while obtaining speech quality and speaker similarity on
  par with the ground-truth audio.
paper_ai_keywords: autoregressive transformer models discrete tokens quantization-based approaches continuous modalities reconstruction quality per-token latent diffusion model zero-shot text-to-speech continuous representations expressive diffusion head image generation variable-length outputs semantic tokens contextual information stopping condition continuous variants discrete speech synthesis techniques comparative analysis discrete baselines continuous speech modeling techniques intelligibility score speech quality speaker similarity ground-truth audio