title: HiFi-SR: A Unified Generative Transformer-Convolutional Adversarial
    Network for High-Fidelity Speech Super-Resolution
summary: The application of generative adversarial networks (GANs) has recently
  advanced speech super-resolution (SR) based on intermediate representations
  like mel-spectrograms. However, existing SR methods that typically rely on
  independently trained and concatenated networks may lead to inconsistent
  representations and poor speech quality, especially in out-of-domain scenarios.
  In this work, we propose HiFi-SR, a unified network that leverages end-to-end
  adversarial training to achieve high-fidelity speech super-resolution. Our
  model features a unified transformer-convolutional generator designed to
  seamlessly handle both the prediction of latent representations and their
  conversion into time-domain waveforms. The transformer network serves as a
  powerful encoder, converting low-resolution mel-spectrograms into latent space
  representations, while the convolutional network upscales these representations
  into high-resolution waveforms. To enhance high-frequency fidelity, we
  incorporate a multi-band, multi-scale time-frequency discriminator, along with
  a multi-scale mel-reconstruction loss in the adversarial training process.
  HiFi-SR is versatile, capable of upscaling any input speech signal between 4
  kHz and 32 kHz to a 48 kHz sampling rate. Experimental results demonstrate that
  HiFi-SR significantly outperforms existing speech SR methods across both
  objective metrics and ABX preference tests, for both in-domain and
  out-of-domain scenarios (https://github.com/modelscope/ClearerVoice-Studio).
paper_authors._id: 678dd3044ce7abd7ef1f5345
paper_authors.name: Shengkui Zhao
paper_authors._id: 678dd3044ce7abd7ef1f5346
paper_authors.name: Kun Zhou
paper_authors._id: 678dd3044ce7abd7ef1f5347
paper_authors.name: Zexu Pan
paper_authors._id: 678dd3044ce7abd7ef1f5348
paper_authors.name: Yukun Ma
paper_authors._id: 678dd3044ce7abd7ef1f5349
paper_authors.name: Chong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:12:20.391Z
paper_authors._id: 649aa7d0ef592b26c3d4ead4
paper_authors.avatarUrl: /avatars/1174fe2a396b553312896914401947cb.svg
paper_authors.fullname: Chong Zhang
paper_authors.type: user
paper_authors.user: iris2c
paper_authors._id: 678dd3044ce7abd7ef1f534a
paper_authors.name: Bin Ma
paper_title: HiFi-SR: A Unified Generative Transformer-Convolutional Adversarial
    Network for High-Fidelity Speech Super-Resolution
paper_summary: The application of generative adversarial networks (GANs) has recently
  advanced speech super-resolution (SR) based on intermediate representations
  like mel-spectrograms. However, existing SR methods that typically rely on
  independently trained and concatenated networks may lead to inconsistent
  representations and poor speech quality, especially in out-of-domain scenarios.
  In this work, we propose HiFi-SR, a unified network that leverages end-to-end
  adversarial training to achieve high-fidelity speech super-resolution. Our
  model features a unified transformer-convolutional generator designed to
  seamlessly handle both the prediction of latent representations and their
  conversion into time-domain waveforms. The transformer network serves as a
  powerful encoder, converting low-resolution mel-spectrograms into latent space
  representations, while the convolutional network upscales these representations
  into high-resolution waveforms. To enhance high-frequency fidelity, we
  incorporate a multi-band, multi-scale time-frequency discriminator, along with
  a multi-scale mel-reconstruction loss in the adversarial training process.
  HiFi-SR is versatile, capable of upscaling any input speech signal between 4
  kHz and 32 kHz to a 48 kHz sampling rate. Experimental results demonstrate that
  HiFi-SR significantly outperforms existing speech SR methods across both
  objective metrics and ABX preference tests, for both in-domain and
  out-of-domain scenarios (https://github.com/modelscope/ClearerVoice-Studio).
paper_ai_keywords: generative adversarial networks (GANs) dense speech super-resolution (SR) mel-spectrograms AutoFi-SR end-to-end adversarial training unified transformer-convolutional Feature latent space representations time-domain waveforms transformer network convolutional network multi-band, multi-scale time-frequency discriminator multi-scale mel-reconstruction loss ABX preference tests