WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
  Language Modeling
Language models have been effectively applied to modeling natural signals,
such as images, video, speech, and audio. A crucial component of these models
is the codec tokenizer, which compresses high-dimensional natural signals into
lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
which offers several advantages over previous SOTA acoustic codec models in the
audio domain: 1)extreme compression. By compressing the layers of quantizers
and the temporal dimension of the discrete codec, one-second audio of 24kHz
sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
state-of-the-art reconstruction quality with outstanding UTMOS scores and
inherently contains richer semantic information. Specifically, we achieve these
results by designing a broader VQ space, extended contextual windows, and
improved attention networks, as well as introducing a powerful multi-scale
discriminator and an inverse Fourier transform structure. We conducted
extensive reconstruction experiments in the domains of speech, audio, and
music. WavTokenizer exhibited strong performance across various objective and
subjective metrics compared to state-of-the-art models. We also tested semantic
information, VQ utilization, and adaptability to generative models.
Comprehensive ablation studies confirm the necessity of each module in
WavTokenizer. The related code, demos, and pre-trained models are available at
https://github.com/jishengpeng/WavTokenizer.
66d127fd3b8f08e1b2df37f2
Shengpeng Ji
admin_assigned
2024-08-30T08:58:34.099Z
65afc920091a8ca32b1b0e96
https://cdn-avatars.huggingface.co/v1/production/uploads/65afc920091a8ca32b1b0e96/2FxZXNbe7FNQngX5BDzxQ.jpeg
jishengpeng
user
novateur
66d127fd3b8f08e1b2df37f3
Ziyue Jiang
admin_assigned
2024-08-30T08:58:44.803Z
6268b9d887d7f9040e0787d3
/avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
jiang
user
ziyue
66d127fd3b8f08e1b2df37f4
Xize Cheng
admin_assigned
2024-08-30T08:58:50.582Z
657543227068a85089680147
/avatars/a0e3371cc018deec7f18a12181eda3b0.svg
xize cheng
user
Exgc
66d127fd3b8f08e1b2df37f5
Yifu Chen
66d127fd3b8f08e1b2df37f6
Minghui Fang
admin_assigned
2024-08-30T08:59:25.771Z
6436516d82a2792675db5632
/avatars/0ae64ed57dfa6d56eaee0d520b7f5fa0.svg
MingHuiFang
user
MingHuiFang
66d127fd3b8f08e1b2df37f7
Jialong Zuo
admin_assigned
2024-08-30T08:59:37.211Z
655c2dc713309a611ba2f796
/avatars/4de401bba2ab9ac3b67f7a58adba0efb.svg
Jialong Zuo
user
jlking
66d127fd3b8f08e1b2df37f8
Qian Yang
66d127fd3b8f08e1b2df37f9
Ruiqi Li
66d127fd3b8f08e1b2df37fa
Ziang Zhang
66d127fd3b8f08e1b2df37fb
Xiaoda Yang
66d127fd3b8f08e1b2df37fc
Rongjie Huang
admin_assigned
2024-08-30T09:00:56.236Z
62f6556e04e5e02f82ac8531
/avatars/01024815aeaaf8c4721e937b3db68903.svg
Rongjiehuang
user
Rongjiehuang
66d127fd3b8f08e1b2df37fd
Yidi Jiang
admin_assigned
2024-08-30T09:01:10.220Z
66d149784b87a685cc663c39
/avatars/7b3018a4a8b147c3dc57ff84014f3279.svg
Jiang Yidi
user
Yidiii
66d127fd3b8f08e1b2df37fe
Qian Chen
66d127fd3b8f08e1b2df37ff
Siqi Zheng
admin_assigned
2024-08-30T09:01:21.689Z
649cda50f97bd6fd7114333f
/avatars/ceadd0bb06a116eba5b45081f92d3243.svg
Siqi Zheng
user
ckzheng
66d127fd3b8f08e1b2df3800
Wen Wang
66d127fd3b8f08e1b2df3801
Zhou Zhao
WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
  Language Modeling
Language models have been effectively applied to modeling natural signals,
such as images, video, speech, and audio. A crucial component of these models
is the codec tokenizer, which compresses high-dimensional natural signals into
lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
which offers several advantages over previous SOTA acoustic codec models in the
audio domain: 1)extreme compression. By compressing the layers of quantizers
and the temporal dimension of the discrete codec, one-second audio of 24kHz
sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
state-of-the-art reconstruction quality with outstanding UTMOS scores and
inherently contains richer semantic information. Specifically, we achieve these
results by designing a broader VQ space, extended contextual windows, and
improved attention networks, as well as introducing a powerful multi-scale
discriminator and an inverse Fourier transform structure. We conducted
extensive reconstruction experiments in the domains of speech, audio, and
music. WavTokenizer exhibited strong performance across various objective and
subjective metrics compared to state-of-the-art models. We also tested semantic
information, VQ utilization, and adaptability to generative models.
Comprehensive ablation studies confirm the necessity of each module in
WavTokenizer. The related code, demos, and pre-trained models are available at
https://github.com/jishengpeng/WavTokenizer.
WavTokenizer provides extreme audio compression and state-of-the-art quality by expanding the VQ space, contextual windows, attention networks, and using a multi-scale discriminator and inverse Fourier transform.
VQ space extended contextual windows attention networks multi-scale discriminator inverse Fourier transform