title: Efficient Speech Language Modeling via Energy Distance in Continuous
    Latent Space
summary: We introduce SLED, an alternative approach to speech language modeling by
  encoding speech waveforms into sequences of continuous latent representations
  and modeling them autoregressively using an energy distance objective. The
  energy distance offers an analytical measure of the distributional gap by
  contrasting simulated and target samples, enabling efficient training to
  capture the underlying continuous autoregressive distribution. By bypassing
  reliance on residual vector quantization, SLED avoids discretization errors and
  eliminates the need for the complicated hierarchical architectures common in
  existing speech language models. It simplifies the overall modeling pipeline
  while preserving the richness of speech information and maintaining inference
  efficiency. Empirical results demonstrate that SLED achieves strong performance
  in both zero-shot and streaming speech synthesis, showing its potential for
  broader applications in general-purpose speech language models.
paper_authors._id: 682cc6e7cb2128493cbe31b0
paper_authors.name: Zhengrui Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-21T08:42:41.951Z
paper_authors._id: 64896496ffac083138eef1e9
paper_authors.avatarUrl: /avatars/216167afeb302896511deb093dbf8700.svg
paper_authors.fullname: Zhengrui Ma
paper_authors.type: user
paper_authors.user: Paulmzr
paper_authors._id: 682cc6e7cb2128493cbe31b1
paper_authors.name: Yang Feng
paper_authors._id: 682cc6e7cb2128493cbe31b2
paper_authors.name: Chenze Shao
paper_authors._id: 682cc6e7cb2128493cbe31b3
paper_authors.name: Fandong Meng
paper_authors._id: 682cc6e7cb2128493cbe31b4
paper_authors.name: Jie Zhou
paper_authors._id: 682cc6e7cb2128493cbe31b5
paper_authors.name: Min Zhang
paper_title: Efficient Speech Language Modeling via Energy Distance in Continuous
    Latent Space
paper_summary: We introduce SLED, an alternative approach to speech language modeling by
  encoding speech waveforms into sequences of continuous latent representations
  and modeling them autoregressively using an energy distance objective. The
  energy distance offers an analytical measure of the distributional gap by
  contrasting simulated and target samples, enabling efficient training to
  capture the underlying continuous autoregressive distribution. By bypassing
  reliance on residual vector quantization, SLED avoids discretization errors and
  eliminates the need for the complicated hierarchical architectures common in
  existing speech language models. It simplifies the overall modeling pipeline
  while preserving the richness of speech information and maintaining inference
  efficiency. Empirical results demonstrate that SLED achieves strong performance
  in both zero-shot and streaming speech synthesis, showing its potential for
  broader applications in general-purpose speech language models.
paper_ai_summary: SLED encodes speech waveforms into continuous latent representations and uses an energy distance objective to model them autoregressively for efficient and accurate speech synthesis.
paper_ai_keywords: speech language modeling speech waveforms continuous latent representations autoregressive modeling energy distance distributional gap residual vector quantization hierarchical architectures zero-shot speech synthesis streaming speech synthesis