title: Scaling Transformers for Low-Bitrate High-Quality Speech Coding
summary: The tokenization of speech with neural audio codec models is a vital part of
  modern AI pipelines for the generation or understanding of speech, alone or in
  a multimodal context. Traditionally such tokenization models have concentrated
  on low parameter-count architectures using only components with strong
  inductive biases. In this work we show that by scaling a transformer
  architecture with large parameter count to this problem, and applying a
  flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
  reach state-of-the-art speech quality at extremely low bit-rates of 400 or
  700 bits-per-second. The trained models strongly out-perform existing
  baselines in both objective and subjective tests.
paper_authors._id: 674dcdbf9e95621d1d73169e
paper_authors.name: Julian D Parker
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:09:30.531Z
paper_authors._id: 657c2cf25c6f0b1f36077033
paper_authors.avatarUrl: /avatars/af538ee6d185d49efc0b7cf8660217ad.svg
paper_authors.fullname: Julian Parker
paper_authors.type: user
paper_authors.user: julian-parker
paper_authors._id: 674dcdbf9e95621d1d73169f
paper_authors.name: Anton Smirnov
paper_authors._id: 674dcdbf9e95621d1d7316a0
paper_authors.name: Jordi Pons
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:09:59.623Z
paper_authors._id: 6682a312d17f4c29394f6c97
paper_authors.avatarUrl: /avatars/18f943e694ce54c4949d467d73cfba14.svg
paper_authors.fullname: Jordi Pons
paper_authors.type: user
paper_authors.user: jordipons
paper_authors._id: 674dcdbf9e95621d1d7316a1
paper_authors.name: CJ Carr
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:10:06.880Z
paper_authors._id: 64932b4a7ed33d93550902bb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RAF-iCFqukRnjZ1VbtVZU.jpeg
paper_authors.fullname: CJ Carr
paper_authors.type: user
paper_authors.user: cjus13
paper_authors._id: 674dcdbf9e95621d1d7316a2
paper_authors.name: Zack Zukowski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:10:13.463Z
paper_authors._id: 666431010875aaaa91b2f1cb
paper_authors.avatarUrl: /avatars/52b4941a8fdfabe7058d6d7f0e4909e9.svg
paper_authors.fullname: Zack Zukowski
paper_authors.type: user
paper_authors.user: drawordoodle
paper_authors._id: 674dcdbf9e95621d1d7316a3
paper_authors.name: Zach Evans
paper_authors._id: 674dcdbf9e95621d1d7316a4
paper_authors.name: Xubo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-02T22:10:28.911Z
paper_authors._id: 64c260105f9e98008d498081
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
paper_authors.fullname: Xubo Liu
paper_authors.type: user
paper_authors.user: Xubo-Liu
paper_title: Scaling Transformers for Low-Bitrate High-Quality Speech Coding
paper_summary: The tokenization of speech with neural audio codec models is a vital part of
  modern AI pipelines for the generation or understanding of speech, alone or in
  a multimodal context. Traditionally such tokenization models have concentrated
  on low parameter-count architectures using only components with strong
  inductive biases. In this work we show that by scaling a transformer
  architecture with large parameter count to this problem, and applying a
  flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
  reach state-of-the-art speech quality at extremely low bit-rates of 400 or
  700 bits-per-second. The trained models strongly out-perform existing
  baselines in both objective and subjective tests.
paper_ai_summary: Transformers with large parameter counts and flexible Finite Scalar Quantization achieve state-of-the-art speech quality at low bit-rates.
paper_ai_keywords: transformer architecture Finite Scalar Quantization (FSQ) speech quality bit-rates