Scaling Transformers for Low-Bitrate High-Quality Speech Coding
The tokenization of speech with neural audio codec models is a vital part of
modern AI pipelines for the generation or understanding of speech, alone or in
a multimodal context. Traditionally such tokenization models have concentrated
on low parameter-count architectures using only components with strong
inductive biases. In this work we show that by scaling a transformer
architecture with large parameter count to this problem, and applying a
flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
reach state-of-the-art speech quality at extremely low bit-rates of 400 or
700 bits-per-second. The trained models strongly out-perform existing
baselines in both objective and subjective tests.
674dcdbf9e95621d1d73169e
Julian D Parker
admin_assigned
2024-12-02T22:09:30.531Z
657c2cf25c6f0b1f36077033
/avatars/af538ee6d185d49efc0b7cf8660217ad.svg
Julian Parker
user
julian-parker
674dcdbf9e95621d1d73169f
Anton Smirnov
674dcdbf9e95621d1d7316a0
Jordi Pons
admin_assigned
2024-12-02T22:09:59.623Z
6682a312d17f4c29394f6c97
/avatars/18f943e694ce54c4949d467d73cfba14.svg
Jordi Pons
user
jordipons
674dcdbf9e95621d1d7316a1
CJ Carr
admin_assigned
2024-12-02T22:10:06.880Z
64932b4a7ed33d93550902bb
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/RAF-iCFqukRnjZ1VbtVZU.jpeg
CJ Carr
user
cjus13
674dcdbf9e95621d1d7316a2
Zack Zukowski
admin_assigned
2024-12-02T22:10:13.463Z
666431010875aaaa91b2f1cb
/avatars/52b4941a8fdfabe7058d6d7f0e4909e9.svg
Zack Zukowski
user
drawordoodle
674dcdbf9e95621d1d7316a3
Zach Evans
674dcdbf9e95621d1d7316a4
Xubo Liu
admin_assigned
2024-12-02T22:10:28.911Z
64c260105f9e98008d498081
https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
Xubo Liu
user
Xubo-Liu
Scaling Transformers for Low-Bitrate High-Quality Speech Coding
The tokenization of speech with neural audio codec models is a vital part of
modern AI pipelines for the generation or understanding of speech, alone or in
a multimodal context. Traditionally such tokenization models have concentrated
on low parameter-count architectures using only components with strong
inductive biases. In this work we show that by scaling a transformer
architecture with large parameter count to this problem, and applying a
flexible Finite Scalar Quantization (FSQ) based bottleneck, it is possible to
reach state-of-the-art speech quality at extremely low bit-rates of 400 or
700 bits-per-second. The trained models strongly out-perform existing
baselines in both objective and subjective tests.
Transformers with large parameter counts and flexible Finite Scalar Quantization achieve state-of-the-art speech quality at low bit-rates.
transformer architecture Finite Scalar Quantization (FSQ) speech quality bit-rates