title: WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
    Language Modeling
summary: Language models have been effectively applied to modeling natural signals,
  such as images, video, speech, and audio. A crucial component of these models
  is the codec tokenizer, which compresses high-dimensional natural signals into
  lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
  which offers several advantages over previous SOTA acoustic codec models in the
  audio domain: 1)extreme compression. By compressing the layers of quantizers
  and the temporal dimension of the discrete codec, one-second audio of 24kHz
  sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
  subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
  state-of-the-art reconstruction quality with outstanding UTMOS scores and
  inherently contains richer semantic information. Specifically, we achieve these
  results by designing a broader VQ space, extended contextual windows, and
  improved attention networks, as well as introducing a powerful multi-scale
  discriminator and an inverse Fourier transform structure. We conducted
  extensive reconstruction experiments in the domains of speech, audio, and
  music. WavTokenizer exhibited strong performance across various objective and
  subjective metrics compared to state-of-the-art models. We also tested semantic
  information, VQ utilization, and adaptability to generative models.
  Comprehensive ablation studies confirm the necessity of each module in
  WavTokenizer. The related code, demos, and pre-trained models are available at
  https://github.com/jishengpeng/WavTokenizer.
paper_authors._id: 66d127fd3b8f08e1b2df37f2
paper_authors.name: Shengpeng Ji
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:58:34.099Z
paper_authors._id: 65afc920091a8ca32b1b0e96
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65afc920091a8ca32b1b0e96/2FxZXNbe7FNQngX5BDzxQ.jpeg
paper_authors.fullname: jishengpeng
paper_authors.type: user
paper_authors.user: novateur
paper_authors._id: 66d127fd3b8f08e1b2df37f3
paper_authors.name: Ziyue Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:58:44.803Z
paper_authors._id: 6268b9d887d7f9040e0787d3
paper_authors.avatarUrl: /avatars/5d022d1e1d7f247e94928ef8ad04c613.svg
paper_authors.fullname: jiang
paper_authors.type: user
paper_authors.user: ziyue
paper_authors._id: 66d127fd3b8f08e1b2df37f4
paper_authors.name: Xize Cheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:58:50.582Z
paper_authors._id: 657543227068a85089680147
paper_authors.avatarUrl: /avatars/a0e3371cc018deec7f18a12181eda3b0.svg
paper_authors.fullname: xize cheng
paper_authors.type: user
paper_authors.user: Exgc
paper_authors._id: 66d127fd3b8f08e1b2df37f5
paper_authors.name: Yifu Chen
paper_authors._id: 66d127fd3b8f08e1b2df37f6
paper_authors.name: Minghui Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:59:25.771Z
paper_authors._id: 6436516d82a2792675db5632
paper_authors.avatarUrl: /avatars/0ae64ed57dfa6d56eaee0d520b7f5fa0.svg
paper_authors.fullname: MingHuiFang
paper_authors.type: user
paper_authors.user: MingHuiFang
paper_authors._id: 66d127fd3b8f08e1b2df37f7
paper_authors.name: Jialong Zuo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T08:59:37.211Z
paper_authors._id: 655c2dc713309a611ba2f796
paper_authors.avatarUrl: /avatars/4de401bba2ab9ac3b67f7a58adba0efb.svg
paper_authors.fullname: Jialong Zuo
paper_authors.type: user
paper_authors.user: jlking
paper_authors._id: 66d127fd3b8f08e1b2df37f8
paper_authors.name: Qian Yang
paper_authors._id: 66d127fd3b8f08e1b2df37f9
paper_authors.name: Ruiqi Li
paper_authors._id: 66d127fd3b8f08e1b2df37fa
paper_authors.name: Ziang Zhang
paper_authors._id: 66d127fd3b8f08e1b2df37fb
paper_authors.name: Xiaoda Yang
paper_authors._id: 66d127fd3b8f08e1b2df37fc
paper_authors.name: Rongjie Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T09:00:56.236Z
paper_authors._id: 62f6556e04e5e02f82ac8531
paper_authors.avatarUrl: /avatars/01024815aeaaf8c4721e937b3db68903.svg
paper_authors.fullname: Rongjiehuang
paper_authors.type: user
paper_authors.user: Rongjiehuang
paper_authors._id: 66d127fd3b8f08e1b2df37fd
paper_authors.name: Yidi Jiang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T09:01:10.220Z
paper_authors._id: 66d149784b87a685cc663c39
paper_authors.avatarUrl: /avatars/7b3018a4a8b147c3dc57ff84014f3279.svg
paper_authors.fullname: Jiang Yidi
paper_authors.type: user
paper_authors.user: Yidiii
paper_authors._id: 66d127fd3b8f08e1b2df37fe
paper_authors.name: Qian Chen
paper_authors._id: 66d127fd3b8f08e1b2df37ff
paper_authors.name: Siqi Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-30T09:01:21.689Z
paper_authors._id: 649cda50f97bd6fd7114333f
paper_authors.avatarUrl: /avatars/ceadd0bb06a116eba5b45081f92d3243.svg
paper_authors.fullname: Siqi Zheng
paper_authors.type: user
paper_authors.user: ckzheng
paper_authors._id: 66d127fd3b8f08e1b2df3800
paper_authors.name: Wen Wang
paper_authors._id: 66d127fd3b8f08e1b2df3801
paper_authors.name: Zhou Zhao
paper_title: WavTokenizer: an Efficient Acoustic Discrete Codec Tokenizer for Audio
    Language Modeling
paper_summary: Language models have been effectively applied to modeling natural signals,
  such as images, video, speech, and audio. A crucial component of these models
  is the codec tokenizer, which compresses high-dimensional natural signals into
  lower-dimensional discrete tokens. In this paper, we introduce WavTokenizer,
  which offers several advantages over previous SOTA acoustic codec models in the
  audio domain: 1)extreme compression. By compressing the layers of quantizers
  and the temporal dimension of the discrete codec, one-second audio of 24kHz
  sampling rate requires only a single quantizer with 40 or 75 tokens. 2)improved
  subjective quality. Despite the reduced number of tokens, WavTokenizer achieves
  state-of-the-art reconstruction quality with outstanding UTMOS scores and
  inherently contains richer semantic information. Specifically, we achieve these
  results by designing a broader VQ space, extended contextual windows, and
  improved attention networks, as well as introducing a powerful multi-scale
  discriminator and an inverse Fourier transform structure. We conducted
  extensive reconstruction experiments in the domains of speech, audio, and
  music. WavTokenizer exhibited strong performance across various objective and
  subjective metrics compared to state-of-the-art models. We also tested semantic
  information, VQ utilization, and adaptability to generative models.
  Comprehensive ablation studies confirm the necessity of each module in
  WavTokenizer. The related code, demos, and pre-trained models are available at
  https://github.com/jishengpeng/WavTokenizer.
paper_ai_summary: WavTokenizer provides extreme audio compression and state-of-the-art quality by expanding the VQ space, contextual windows, attention networks, and using a multi-scale discriminator and inverse Fourier transform.
paper_ai_keywords: VQ space extended contextual windows attention networks multi-scale discriminator inverse Fourier transform